Programmable tactile touch screen displays and man-machine interfaces for improved vehicle instrumentation and telematics

ABSTRACT

Disclosed are new methods and apparatus particularly suited for applications in a vehicle, to provide a wide range of information, and the safe input of data to a computer controlling the vehicle subsystems or “Telematic” communication using for example GM&#39;s “ONSTAR” or cellular based data sources. Preferred embodiments utilize new programmable forms of tactile touch screens and displays employing tactile physical selection or adjustment means which utilize direct optical data input. A revolutionary form of dashboard or instrument panel results which is stylistically attractive, lower in cost, customizable by the user, programmable in both the tactile and visual sense, and with the potential of enhancing interior safety and vehicle operation. Non-automotive applications of the invention are also disclosed, for example means for general computer input using touch screens and home automation systems.

This application is a continuation of U.S. patent application Ser. No.11/272,868, filed Nov. 15, 2005 now abandoned, which is a continuationof U.S. patent application Ser. No. 09/789,538, filed Feb. 22, 2001 (nowU.S. Pat. No. 7,084,859), which claims the benefit of U.S. ProvisionalApplication No. 60/234,934 filed Sep. 26, 2000 and U.S. ProvisionalApplication No. 60/183,807 filed Feb. 22, 2000. This application is alsoa continuation-in-part of U.S. patent application Ser. No. 11/495,666,filed Jul. 31, 2006 (now U.S. Pat. No. 7,714,849), which is acontinuation application of U.S. patent application Ser. No. 09/435,854,filed Nov. 8, 1999 (now U.S. Pat. No. 7,098,891), which is acontinuation of U.S. patent application Ser. No. 08/496,908, filed Jun.29, 1995 (now U.S. Pat. No. 5,982,352). The disclosures of the abovepatent applications are hereby incorporated by reference in theirentirety.

BACKGROUND OF THE INVENTION

This invention seeks to dramatically increase the utility of carinformational displays and controls, while at the same time enhancingsafety by improving sensory data presentation and ease of interactionwith vehicle controls and data sources. The programmable nature of thedisclosed devices also creates new methods for how data is delivered andutilized.

The disclosed invention, including co-pending applications incorporatedby reference, contains unique embodiments which allow one to interact,by feel, with a display, called herein a “programmable tactile display”.It encompasses two main focus areas: A display having features commonlyassociated with a touch screen, but in a new form which can be sensed inseveral tactile manners, as well as visually. A tactile selection oradjustment means, such as a knob, slider, or switch, programmable in itstactile or visual nature, and generally operated in conjunction with thetouch screen just described.

These features in turn provide the basis for a automobile instrumentpanel (dashboard) or other control panel which can be operated withoutundue concentration on visually reading the display while working thecontrols. It serves as an alternative, or adjunct, to voice activatedsystems being considered today to allow increased functionality withsafety of vehicle operation. In several embodiments, the force or othersensation felt can itself be programmably changed, adding to driverunderstanding, and enhancing safety.

Because it resembles today's dashboards, and can be used for the basiccontrol functions of the vehicle, the invention provides not only apotential means of telematic connectivity while driving (e.g. with theinternet, cellular telephonic sources or the like), but a much moreuseful display and control system capable of many morefunctions—including the primary vehicle control functions, if desired.

In addition, I feel that a dashboard incorporating the invention can bebuilt at lower cost than a conventional dashboard, especially asvehicles become ever more loaded up with navigational systems and otherelectronic functions incidental to the control of the vehicle.

There is no known prior art having the above characteristics.

SUMMARY OF THE CHALLENGE AND OPPORTUNITY

The automotive dashboard is today a confused array of switches, knobs,dials, gages, and other tactile physical selection or adjustment meansand instruments. It is often hard to see and to understand, and cancause undo distraction to the driver. As a device, it is filled withdifferent parts and thus expensive to manufacture both in serialquantity, and in redesign and tooling for new models. Furthermore, it isinflexible, and invariant once manufactured. It cannot be changed in itsdesign by the user, and cannot be easily changed by the manufacturer orthe dealer.

In addition, the instrument panel in its current form it is at its limitin so far as its ability to present data which can be acted on. Thereisn't any more room on the dash to put more devices, and various safetyissues such as airbag deployment and passive interior safety precludemany choices. Data which might be desired for action can be from thevehicles own controls and state, its surroundings such as from othervehicles or parked objects, or could include material downloaded fromthe internet, or navigational data from satellites.

Because of this, Voice recognition techniques are now being researchedin earnest in order to allow the user to interact with computer basedfunctions. IBM VIA VOICE and DRAGON SOFTWARE Naturally Speaking productshave already reached the general office market, and are reasonablyaccurate and effective if the surrounding environment is quiet andstable. But in the car, this is not the case, and even specializedlimited capability Telematic systems such as Fonix CorporationsAutomatic Speech Recognition (ASR), L and H, etc, have lots ofproblems—for example:

Noise—the car is not an ideal environment under any circumstance, nevermind with CD Player at full volume;

Different drivers, with different accents, practices;

Limited or no teach in available to perfect the function of therecognition program; and

Passenger disturbance, and conversation interruption (i.e. a backgroundnoise source which disturbs the recognition, or the actual intonation ofthe speech.

These problems result in function which is possibly error prone or timedelayed. Having to repeat or worse, having the wrong action, is notdesirable in mission critical driving situations so to speak. The Cellphone problem is notorious. In the extreme, a form of rage can take overif the system frustrates the driver. For this reason, voice recognitionsystems used for actual vehicle functions (as opposed to dialing phonesand the like) have in effect, backup controls—defeating the concept ofincreasing dashboard space, and control comprehension.

In addition there is a sociological problem—talking to ones car inunacceptable to many, even if the voice recognition program works. Forsome this is always true, for others its true with passengers present.

In addition, the presentation of data by the computer to the driver bytext to speech programs is known to have unacceptable intonation or timelatency in many circumstances by the average user, and in any case iswoefully inadequate for graphical or tabular information, for example.The data to be presented to the user, even in a simple list of choices,has to be vocally spoken by the computer to the driver in sequence,taking a great deal of concentration to avoid missing an importantpoint, and taking minutes to do what can visually be done in a glance.

Voice aside, perhaps the only way more data can be presented to users(driver, and/or passengers), who might wish to engage in internet orcellular activity while driving, for example, is through the use ofprogrammable displays, and programmable entry devices, which can bevaried to suit the need of the user at the instant of use. Such displaysare commonplace today in the computer world (e.g. “Windows”, “WebBrowsers”), so why not in a car?

It is noted that unbridled use of such displays by the driver is howeverlimited by laws in several states as for example pointed out in U.S.Pat. No. 5,949,345 owned by MICROSOFT corporation, which desirablycovers an operating system for the computer driving the display to allowlegal operation. Also germane are Ser. No. 08/564,586 and Ser. No.08/668,781 assigned to MICROSOFT.

One problem is how to interact with the display, without having akeyboard—which is generally too cumbersome, switch filled, and spaceconsuming for a car dash, armrest, or other interior location. And amouse is pretty much impossible as well.

Accordingly, some companies have proposed or developed conventionalchangeable computer window type systems using up/down/left/right buttonsor even joysticks to select the screens and menus thereon in an attemptto solve this problem. Such devices are however, not intuitive and hardto use, for at least 95% of the populace I feel. Because of suchproblems, even where implemented, they are relegated to only noncritical functions, such as navigation, climate control, cell phonedialing or audio system entertainment.

If such computer based systems could be used by the general populace,without the limits of voice recognition, a very large application areacomprising in its limit the total function of the vehicle could result.In addition, improved productivity could result as people in the USAspend over 500 million hours it is said in their cars, and much of thistime is wasted, from a business point of view. Even to salvage 1% is 50million hours, which corresponds to over a billion dollars saved.

I feel a key part of the answer lies in a form of tactile display ortouch screen with tactile properties not hereto fore seen. This displaycould, depending on its construction, occupy some, or even most of thedashboard (also called dash, or instrument panel). Alternatively and orin addition, it could occupy the region in the center of the car calledthe “center stack”. It should also be noted that the problem may alsorequire a multi-sensor solution, which will be discussed herein.

Some requirements of an Automobile display/touch screen could arguablybe at least the following:

1. Sturdy—and conducive to interior passive safety, and regulations

2. Should allow multiple functions to be undertaken simultaneously,either all by feel, by glance and feel, or by combinations oftouch/feel, body positions, voice and other variables.

3. Should provide enhanced measure of safety by requiring lessconcentration on controls than present with many controls today.

4. Allow programmable screen operation by feel if possible, for ease ofuse while driving (or while engaged in other tasks) and safety. The“feel” sensation should be itself programmable, such that varyingresponses can be given the operator, depending on what screen and whatfunction or what amount is being actuated or interrogated (that is youtouch it to get a sensation first indicative of the current value orscreen type)

5. Activate functions easily

6. Not affected by environmental issues, such as ambient light,contamination

7. Not sacrifice airbag function—thus must not be where the airbag is,or the airbag/and screen module must be of another design (one of whichis disclosed herein)

8. High resolution, to allow graphics and dense information such as thatdownloaded from the internet to be displayed, when desired.

9. It should desirably respond to z inputs (into the display screenitself) to give added dimension to function. (e.g. volume of sound, air,etc,) or combine a switching function with a turning, sliding or otherfunction (similar to the heat knob of a Olds Aurora for example). Andresponse to inputs at angles to the screen normal would also bedesirable in some cases.

Prior Art

Directly to the point, there are no touch screens known to me, otherthan my own, which have a physical tactile “feel” aspect to them—afeature which I consider essential for the vehicular control applicationof touch screens. Having said this, a review of conventional touchscreen art for automobiles follows.

Where limited functionality is all that is needed, conventional touchscreens of limited size are an answer. To my knowledge, the applicationof touch screens for computer input in automobiles was first achieved inthe 1988, in the General Motors Buick Riviera using a relatively smallCRT display located in the center of the dashboard. Such a display isdescribed in U.S. Pat. No. 4,787,040 by Ames, et al. entitled Displaysystem for Automotive Vehicle issued to IBM Corporation Nov. 22, 1988.

More recently, numerous manufacturers have done the same, for thepurpose of navigation, using instant GPS data and stored maps. However,none of these displays is easy to use by the driver, being small,difficult to interact with physically, and out of the general line ofsight of the driver. This is a problem as one has to see the screen totouch the box corresponding to the input function desired. And to seethe screen, you have to take your eyes off the road for a significanttime. There is no physical sensation associated with the function whichwould allow you to actuate the function “by feel”. Indeed because ofthis, many of the GPS systems being sold, are just displays, withfunction selection and the like achieved with the standard assortment ofbuttons, switches, dials, etc. And where touch boxes on screens areused, they by necessity have limited functions.

Other problems with computer displays of the multiple windows type existwhen these are used in vehicles—this is discussed in U.S. Pat. No.5,995,104 by Kataoka, also aimed at a Navigation system. The instantinvention can use such windows in the touch screens it provides, butpreferably uses more physical “objects” representing typical dashboardinstrumentation.

Finally, as another example of display locations in vehicles, a recentU.S. Pat. No. 5,871,251 assigned to Prince Corp) discloses an LCDdisplay located in the sun visor, for use by the driver, having no inputcapability which would allow the driver to make use of it in normaloperation (even if he could see it well).

Another problem, is that the touch screen gets really attractive, as youenvision it as a replacement for a large measure of the automobile userinstrumentation and control functions of today, a dramatic move, filledwith other advantages in flexibility, ease of use and user benefits—pluspotentially lower cost.

What about then, a touch screen of the LCD type, with a surface acousticwave (SAW) based touch cover screen? (e.g. Microtouch brand). One couldjust put one of these across the whole dashboard, at least in sections.SAW types do not have some objectionable properties of cover materialsand the like which can get damaged which some other types ofconventional screens possess.

This use of a conventional LCD based (or alternatively plasma display orother type) flat panel touch screen has several problems (and is veryexpensive today). These are:

The requirement to see the function touched, with no physical feedbackfrom doing so, as pointed out above;

A potential problem in large size with complying with interior passivesafety regulations (should a persons head hit it, say);

A problem in coexisting physically, and functionally, with dashboard orsteering wheel installed airbags (also called SRS or supplementalrestraint systems) and their actuation;

A potential problem with durability, being able to survive hits by kids,feet on the dash, etc., as well as degradation of performance bycontaminants (a major problem of conventional technologies recentlyaddressed in U.S. Pat. No. 5,923,319 Bishop, et al; and

Possible thermal drift and resolution problems due to the extremetemperature situations in vehicles.

The safety related issues are a significant stumbling block, but theease of use without undue concentration issues are as well, and apply toother fields besides—indeed anywhere a touch screen would be desirablebut not easy to use by someone who can not take the time to look, or towhom looking is inconvenient or uncomfortable, or in the case of thehandicapped, difficult or impossible.

It is also noted, that the objectionable requirement to touch what onesees (and thus take ones eyes significantly off the road in many cases),has undoubtedly led manufacturers of the most recent set of navigationalscreens, to use a fixed set of keys as inputs, with the data displaybeing the only thing programmable. In this case the person can feel theswitch, and only quickly glance at the screen for data once presented.But the keys and switches used are by necessity small, and cluttered.

To conclude, conventional touch screens can work in vehicles, but intheir form heretofore they have no tactile properties. Thus today theyneed to be augmented by voice recognition and/or tactile devices placednear them if they are to be used for important functions of the vehicle.

SUMMARY OF THE INVENTION

The instant invention disclosed herein seeks to provide a vast increasein flexibility of data presentation and manipulation, while obviatingthe problems of conventional technology, and doing so at a low priceencouraging wide spread automotive application. (indeed it is thought tobe less expensive to manufacture, and much less expensive to design andredesign, than a conventional dash with its various Input/Outputdevices). While the focus is on vehicular application, it is noted thatthe invention is not limited thereto.

Safety is enhanced by providing a potentially safer car interior and onemore friendly to elderly or handicapped drivers and those visuallychallenged (primarily due to the larger displays and control devicesmade practical, while at the same time having more functionality). Inaddition, it can enhance safety by promoting a degree of dashstandardization between vehicles not now practical from a marketingpoint of view. This is because certain functions could be common, withthe decorative and other portions customizable.

The invention is also differentiated in that it expands the stylisticand after market business opportunities associated with vehicle sales.Just one example disclosed is a single use or mission specificaccessories to enable safe driving while executing detailed specializedtasks, for example a project worked on ones way to work, or a trip takento a specific place.

There are many embodiments of the invention disclosed herein. Oneversion, has a major advantage for near term commercialization in thatit results in a dashboard and/or center stack which is as close aspossible to today's cars while possessing a high degree of programmablyvariable operation needed for enhanced use by a driver.

It is especially noted that the invention serves to eliminate problemsbrought about by reliance on voice communication between driver andvehicle, and telematic connections to same. Voice recognition hasproblems, as it often requires prompts for the driver, which are oftentime consuming, frustrating, and difficult to hear. Undue concentrationon same can also cause unsafe driving (for example in the presence ofsirens). In addition, voice is linear, and sequential. Only onemessage/action can be executed at once. And the whole message has to beunderstood by the driver, or vehicle or telematic connection—or all canbe lost and the process required to be repeated, which can providefrustration and endanger safety in extreme circumstances.

With the instant invention however, key functions can all be implementedwith little or no voice input/output by the driver at all. And more thanone function can be done at once, using multiple fingers, hands, headmovements and a variety of alternative input mechanisms.

The invention also contributes to safe operation in other ways. Forexample the dash can be built with a lack of protruding devices ifdesired, contributing to passive safety. In addition, the invention isalso unique among touch screens, as it allows the touch screen tocoexist so to speak, with an airbag. This in-turn may allow novellocation of the invention in the steering wheel. This is the easiest touse of any location, as the drivers hands are normally on the wheel.

Another good location for the driver, also largely unexploitedheretofore is in the center console (also called center stack). Even thearmrest can be used, desirably assisted by a heads up display on thewindshield. Other embodiments of the invention describe displaylocations of use to passengers, or of use to the driver on an periodicbasis only.

Since the technology disclosed herein and in co-pending applicationsincorporated by reference is in many ways universal in its application,some applications are herein disclosed of use in the home and businessas well. These often do not require “feel”, but benefit from theinvention's similarity to familiar tactile physical selection oradjustment means found in the home or workplace—knobs, switches, levers,sliders, and the like.

The basic aspects of some features and/or embodiments of the inventionhave been disclosed in provisional patent applications, co-pendingapplications and patents incorporated herein by reference. These are nowbriefly summarized.

Co-pending application Ser. No. 09/435,854 by the inventor (and recentlygranted U.S. patents related thereto) describes the use of unique touchdevices for automotive dashboard related applications, and in addition,discloses new type of data entry device for computers and otherelectronic equipment generally in the category of digitizers and touchscreens having several unique properties. This device, in the formusable here, is most generally based on the use of one or more TVcameras to view optically discernable data associated with the screenfrom the rear in order to determine the location in screen coordinates(and z in some cases as well) of one or more touch points. Primarilydiscussed therein are devices which use the electro-opticaldetermination of temporary surface distortion caused by the physicalinput signal, or force, creating distortion (e.g. a finger “touch”).This is referred to as “surface distortion imaging” and depends on theability to detect, and in some cases quantify, small distortions of orstresses in member over a large (by comparison) area. This technologyhas some unique advantages for automotive control and displayapplications, the subject of this disclosure, which elaborates on theversions shown previously. It is also highly useful for otherapplications, such as Kiosks, control panels in the home, and the like.And further disclosed are means to allow tactile physical selection oradjustment means such as knobs, levers and switches to be integratedwith a touch screen, an area expanded on further in this application.

In addition to the unique touch screen just described, other referencedpatent applications by the inventor incorporated by reference describenovel means to determine other events which cooperatively orindividually may a be imputed to a computer using electro-opticallydeterminable datum's on persons or other entry means, typically objectssuch as steering wheels, switches, and the like in a vehicle. Includedis data input using direct camera observation of the user, or datum'sattached to the user, or objects held by the user. With the advent oflow cost cameras, the use of such camera systems becomes practical forautomotive interior application. And disclosed herein are other touchscreen transduction methods which can be used alternatively or incombination with those of previous applications.

The above features are very useful in answering the needs of potentialapplications in cars, trucks and other vehicles, as well as for certainother instrumentation and related applications—even in the home. Inparticular, the invention herein, even beyond previous disclosures willbe further shown to have unique advantages in the following areas.

1. Feed back to the user of touch related activities

2. Safety enhancement and compatibility with existing safety systems andnorms.

3. Ruggedness and ease of application in difficult environments

4. Novelty in terms of the benefits for the user

5. New business methods resulting from the data intensive activitiesmade possible in the car

6. Novel locations made possible for the screens

7. Additional novel touch screen sensing approaches including thoseusing TV cameras to view the user or objects.

ADVANTAGES OF THE PRESENT INVENTION

The instant invention is aimed primarily, but not necessarily for dashmounted touch screen displays, which can literally fill the dash ifdesired, or at least provide a large display at any or all of theselocations of primary interest:

In the center of the dash, or in the console (also called center stack)for use by driver and/or passenger; Replacing the existinginstrumentation cluster in front of the driver; Located in the steeringwheel, in addition to or in place of said cluster; and In front of thepassenger.

Safety Advantages Include:

Reduction in sources of anger or frustration caused by alternative voicebased systems; Large easy to read data—much larger than possible today,and ideally suited for just quick glances and elderly drivers, anddrivers in general who are far sighted and don't have close up glasses.The large size is due to the ability to time share the dashboard spaceIn addition the Focus of eye can be on objects outside the vehicle, asletters big enough to read even if out of focus;

Can lead a degree of dash standardization between vehicles not nowpractical from a marketing point of view. One can in an extreme, carryyour own dash program with you from one vehicle to another (e.g. arental car)!

Much more data available, in a wide expanse of screen. This alsoincludes TV imagery of the surroundings of the vehicle (eliminatingblind spots, aiding parking, etc). Novel too is the ability to provide ahuman assist, e.g. designation, to automatic vision and ir systems;

The tactile control of the computer system used can be customized notonly by the user in general, but for specific purposes—what I call a“Mission specific” tactile and visual input and output. This makes itstill safer by providing just the right tactile controls for theapplication at hand. Some exemplary versions can potentially becollocated with airbags, allowing placement in positions most easy touse, such as the steering wheel.

It is in addition, a goal of the invention to improve vehicle safety by:

Providing an instrument panel which has larger and easier to read andcomprehend controls;

Providing a means to inform drivers of hazardous conditions;

Providing a means to automatically cause the vehicle to take action inhazardous situations;

Providing a means for drivers to seek important safety relatedinformation;

Providing a dashboard of enhanced passive safety; and

Providing a means for telematic connectivity which is conducive to safedriving.

It is a also a goal of the invention to make possible the safe use ofaesthetically pleasing large screen programmable displays by a driver asthe primary means of data input and output. And to provide means forchanging these displays and inputs to suit the needs and desires of theuser.

It is an added goal of the invention to provide tactile displays whichallow a driver to “feel” the information desired as well as see it,similar in many ways to instrument panels of today but with many addedbenefits.

It is a goal of the invention to provide a touch screen or other tactiledisplay having a passive or active tactile feedback to a users touch.

It is another goal of the invention to provide a method by making touchcommands usable with just a glance from a driver to the region of alarge easy to read display.

It is a further goal to provide displays and actuation devices forvehicle functions which can be customized at a factory or dealership orotherwise (for example from a web site).

It is also a goal of the invention to provide displays and inputs notjust for the functions of the vehicle operation systems, but for amyriad of added functions, such as navigation, reconnaissance,communication, information download and retrieval, gaming, and evenpleasing interior design and changes thereto. Many of these functionsalso can serve applications outside the vehicle as well, such as in thehome or workplace.

It is another goal of the invention to provide a touch screen havingcontrols which indicate to the user via physical touch sensible signals,such as acoustic waves, a condition, such as the control device touched,the particular selection made of choices concerning same (e.g. heat orwipers), or the magnitude of the setting desired of the choice made.This can also indicate what the current value of the choice or functionis at the time of touch, not just changes therein.

It is a goal of the invention to further provide means by which complexinstructions for use can be presented in easily readable form co-locatedwith the control function whose instruction is needed.

It is a goal of the invention to provide means by which multiple inputsand outputs (e.g. touch, voice and direct optical viewing of human orhuman actuated switch positions) can be used in concert.

It is a goal of the invention to provide means for both z (directioninto the display screen) and force based input of single or multiplehuman or object inputs.

It is goal of the invention to provide a new form of flat panel touchscreen having tactile relation of user touch to the screen.

It is goal of the invention to provide a flat panel display havingtactile physical selection and adjustment capability.

It is goal of the invention to provide a touch screen which opticallyprovides feed back to a driver in his line of sight of touch screen ortouch pad input movements.

It is goal of the invention to provide a device, which may also includea touch screen, equipped with interchangeable front panels which havespecialized and if desired, purpose special physical devices forinteraction with the sensing arrangement used.

It is goal of the invention to provide a touch screen having a displayand associated control function which moves around the point of touch,using for example a vector of touch, as opposed to requiring a person'stouch to track a predetermined fixed display a task requiring much morevisual concentration.

It is a goal of the invention to provide methods for using a touchscreen for fine manipulation of data on the screen, heretofore notpossible.

It is goal of the invention to provide a method for controlling avehicle or other device which uses a combination of functions,particularly touch, voice and gestures.

It is another goal of the invention to provide for methods of workimprovement in use of computer tools such as spread sheets, wordprocessing, computer aided design and the like.

It is a further goal of the invention to provide improved controls forautomation systems, including those in the home.

It is noted that this invention presents fundamentally differentconcepts for touch screens that those which are menu driven as known inthe art (though it can use those too). It calls for “Natural” functionsmore related to the existing instrumentation of the vehicle and theexperience base of the driver, including physical implementation notknown heretofore in the touch screen or control art, whether for cars orfor any other purpose.

These tactile methods, together with novel screen embodiments and othermeans for communication with automotive and other control functions arenow disclosed.

While described as displays or “screens” in typical computer parlance,there are in general, combinations of programmable displays, tactiletouching aspects and gesture and voice, to make a synergistic whole.Also the screens can be as noted in front of the driver or passenger, orin the center stack (middle region of the dash) for example, as desired.They are discussed herein in the center stack as this is thought thefirst point of application due to non-interference with airbag locationand other factors.

Finally, it is possible that the disclosed display and instrumentationinvention will prove safer for vehicle occupants than those of today.One reason is that protruding switches, knobs, stalks and the like canbe eliminated, and these in some cases have caused injury, particularlywhen passengers have been thrown into the center part of the vehiclewhich has no protective airbag today.

BRIEF DESCRIPTION OF FIGURES

The invention will further be described in the following figures:

FIG. 1 Illustrates conventional automotive practice, and arrangements ofscreens and input devices (touch or otherwise) which comprise variousautomotive operative control and information embodiments of theinvention. Further illustrated is a preferred embodiment for near termcommercialization, located in this example in the center stack. Thisembodiment discloses programmable tactile physical selection oradjustment means and is most like the dash operation in vehicles oftoday, but in addition has programmable visual and tactile aspects ofhigh utility. Projection and LCD display examples with different typesof optical sensing are illustrated, as are interchangeable tactileelements.

FIG. 2 illustrates additional detail and features of the preferredembodiment for near term commercialization possessing a physical detailtactile feel, which may be also operated in conjunction with the surfacedistortion or other types of touch screens disclosed herein

FIG. 3 is a basic embodiment of a dashboard located touch screen,similar to that of FIG. 2 of Ser. No. 09/435,854. Also illustrated inFIGS. 3 b to 3 d are various examples of tactile-like manipulation ofdata using such a screen

FIG. 4 a preferred arrangement for safety and other purposes, in whichthe illumination and image projection source are off-axis, such that anairbag (also called supplemental restraint system, or SRS) can bedeployed on-axis. Both optical and computer correction techniques areused to make a presentable display.

FIG. 5 illustrates a touch screen having indented or raised portions ofthe screen or overlays thereon to facilitate the driver finding thecorrect location for input action the touch screen may provide, to aidknob turning, slider moving, switch rocking, turn signaling,transmission mode selection and other usage of various tactile physicalselection or adjustment means.

FIG. 6 illustrates a combined touch, gesture and voice activated system

FIG. 7 illustrates a touch screen on the dash, whose control is further(or alternatively) equipped or enhanced with an optional TV camera basedhuman extremity (head, fingers, hands, etc) tracker in the roof andother features.

FIG. 8 further illustrates methods provision and functionality ofexternal data using a screen with TV images

FIG. 9 illustrates another version, in which the steering wheel andairbag also encompasses the display and touch device.

FIG. 10 illustrates an embodiment of the invention where a Screendisplay moves to or around a touch point touched by a user, using avector of touch or a time based input.

FIG. 11 is an alternate embodiment of the invention using the image of ausers body portion or object, using light from a projection source, orfrom inside the car to back illuminate said portion.

FIG. 12 is an alternate acoustic embodiment of the invention using asurface acoustic wave (SAW) conventional type touch screen or otheracoustic means and employing a LCD Flat panel display

FIG. 13 illustrates a unique “Heads up display” (HUD) of touch screenand touch position, to aid the driver to work the touch screen whiledriving.

FIG. 14 illustrates a novel screen design for distortion based touchscreens. Also illustrated is an Optical touch screen equipped with aforce feedback signal which is acoustically generated by a piezoelectric or other transducer providing an acoustic wave force pulse backto the user to signal that one is close to the point desired or hasreached it for example. It can be pulsed, or of static or varyingrepetition rate or frequency, or of varying amplitude.

FIG. 15 illustrates a display screen responsive to a laser pointer orlike device used by driver or passengers to input data.

FIG. 16 illustrates an embodiment of the invention having externalcameras and cameras to sense the orientation of a drivers head

FIG. 17 illustrates a front sensed switch overlay for conventional orother touch screens.

FIG. 18 illustrates a ball joint or gimbaled physical selection oradjustment means in which position and orientation of the handle inputis determined optically using for example a four point target set.

FIG. 19 illustrates a multipurpose home application of the invention

FIG. 20 illustrates an application for difficult environments, such as ahome shower bath

FIG. 21 illustrates another screen input embodiment, using camerassensing humans or objects through the projection screen of the invention

FIG. 22 illustrates further camera sensed controls of the invention

FIG. 23 illustrates a keyboard embodiment of the invention

FIG. 24 is an touch screen improvement whereby touch designationprecision of small displayed features is assisted by use of two fingersand/or demagnified cursor movement controlled by ones finger movement.

FIG. 25 illustrates further embodiments of the “touch mouse” of FIG. 24using a pinching or other action-in this case with respect toinformation in a spreadsheet cell or a word document word or letter.

FIG. 26 further illustrates a touch mouse using a sliding lever orrotary knob or joystick representation.

FIG. 27 illustrates embodiments having elastic touch screen propertiesand employing photo-elastic and other principles

FIG. 28 illustrates an embodiment of the invention where one firsttouches a screen at any point desired or comfortable, and said touch istemporarily indented into the screen to serve as reference.

EMBODIMENTS OF THE INVENTION

FIG. 1

This invention is predominately aimed at providing a new form ofinstrument panel and related controls, based on displays, includingtouch screen displays, which have tactile attributes of considerableutility for the driver of the vehicle. By themselves, or in concert withvoice techniques, these tactile displays enable the effective selectionof functions in control related applications, not just in the vehicle,but in the home and elsewhere. Many of the disclosed embodiments arealso thought to improve the safety of vehicle operation, while doing soat less manufacturing cost.

In a vehicle, tactile displays of the invention can be located on thepassenger side (mainly for amusement and internet communication), on thedriver's side as a replacement for the existing instrument cluster, orin the middle (also called center stack) which is the easiest toimplement today because of lack of requirement of airbag application andthe fact that the steering wheel does not obstruct hand or eyecommunication with the display screen.

Before providing detail on the specific technology employed, it is ofinterest to consider what sorts of new presentations of data to thedriver and passengers of a vehicle might be, if such technology wereemployed, especially but not necessarily that of the invention. Only ifdisplays having suitable tactile aspects are available (the primary goalof the invention herein), will these possibilities likely ever berealized, and least in the near to mid term.

FIGS. 1 a and b illustrate one example of a conventional Automobileinstrument panel of today, in this case of a SAAB 9-5. This dashincludes many small buttons, displays and controls, and yet has none ofthe features desired for advanced forms of telematics, navigation andother features disclosed herein. It is a good example of a nice designat its limit.

FIG. 1 c Illustrates one example of a conceptual vehicle basic dashboard 1, and center stack 5, with tactile displays and/or touch screens10 14, made possible by this invention, incorporated in The steeringwheel center, 10 The center stack, 11,12 The console, 14 The passengerside of the dash board, 13 And possibly the instrument cluster itselfNot here illustrated.

As will become apparent on consideration of this invention, it ispossible to provide in an automobile dash, a highly effective touchscreen or other display of large size having the necessary tactile feel,passive safety, and air bag accommodation where needed to make itacceptable from a safety point of view.

This unique ability opens up a host of novel dashboard implementationembodiments presented here, which are sometimes also possible lesspreferably with other prior art touch screen devices in certaincircumstances as well. The reason for this is the huge increase inflexibility presented by the tactile touch screen concept, and itsability to greatly increase the active information which may be madeavailable to improve automotive safety, as well as the drivingexperience, at the same time in total synchronism with current trends toinformation availability at all times and places, including in onesvehicle. (a concept referred to as “Telematics”).

Many of these advantages accrue because the same display can function asthe controls, the vehicle operation information presentation, and thetelematic function displays. And on top of that, can provide informationfrom cameras and other inputs directly where needed by the driver (assuch information can temporarily, as in time of stress or crisis,replace some or all of the functional displays, for example of speed orengine RPM).

Some novel screens and other dash configurations and representationsaccording to the invention, which give an idea of the breath ofpossibilities are given below. a. Standard screen b. Highway screen c.City Traffic screen d. Audio enjoyment screen e. Night vision and fogscreen f. A Difficult driving or “Crisis” screen, where all functionsrelate to navigating or recovering from a difficult situation—such asvery heavy traffic, ice conditions, etc. g. Malfunction and alarmscreen, including Automatic switch to a “trouble mode” so to speak whensomething wrong is detected h. Telephoning faxing and Email screen i.Climate control optimization screen j. Exaggerated character and inputscreen (for elderly or infirm persons, or as desired) k. Internet andcommunication screen (passenger) 1. Internet and communication screen(driver moving, and driver stopped) m. A Techno screen, with graphs,charts, etc of interest (laws may prohibit some of these from beingdisplayed to the driver while moving—in this case a set of abbreviatedscreens with “legal information” might just be displayed. n. A“preflight” checkout screen to allow the driver to ascertain correctfunctions and settings of various subsystems o. Preventive maintenancescreen, with data, predicted service times, and even camera images ofkey parts of vehicle such as wheels, engine, etc. p. “Global”navigational screens, including GPS based MAP data as is now prevalentfor guiding one to a destination q. “Local” navigational screen forparking, in which camera and other sensory data (microwave, ultrasound,IR lidar, etc) are greatly magnified and presented. To aid in parking ornavigating in tight quarters. r. Parking spot screen, providing data onlocation of parking spots in for example, a garage, using datadownloaded to the car upon entering or nearing said garage. s. Carsubsystem amusement and education screen, for displaying, for example a.3D representation of airflow, with different HVAC and vent positionsettings, b. 3D representation of sound, with different settings,including if desired the monitoring of ambient sound c. 3Drepresentation of rear and front viewing situations by various opticalsubsystems (mirrors, cameras, IR or other night vision, etc) and radarsubsystems if used.

The above screens can, if desired, occupy 60 70% of the width of thedash, in front of the driver and in the center. Or they can be in frontof the driver, in the instrument cluster area, or on the steering wheelas disclosed herein. They are primarily for use by the driver,unassisted, but clearly have use for passengers as well. If they are notfor the driver, many of the comments regarding vehicle functions may notapply (e.g. the need for a speedometer or other critical car function ifany to be displayed somewhere at all times may not be needed).

These screens do not represent the totality of novel functions which maybe provided, but are ones that I feel are of considerable utility.Generically, these novel screens have one or more of these commonthreads. 1. An ability to switch automatically to a new safety orientedstate on input of data from sensors which indicate such is called for.This state can be activated manually as well 2. an ability to switchamong several screens 3. an ability to be switched from one to another,as well has have individual functions triggered from a variety of humancommand inputs, including touch related activities of various sorts,plus hand motions, head motions, finger motions, and the like. 4.

Some more detail on the screens listed above appears in a sequence ofFIGS. 29 a to 29 s

FIG. 1 d illustrates a preferred highly tactile center stack embodimentfor near term commercialization, as it is closest to the instrumentpanel of today in terms of touch and feel, and visually, yet offers thefull programmability required tomorrow and it is also thought to promotesafer driving. The overall shape, in this example, is that of a SAAB9000 and 9-5 which, like the Oldsmobile Aurora has a portion of thedash/center stack as its called, canted toward the driver, representingan almost ideal location for a touch screen or other embodiment of theinvention.

The base screen functions are depicted in the quasi rectangular portionof the center stack area in question, in keeping with the notion of arelatively simple projection display on a scattering screen in the samearea.

This embodiment utilizes “classical” tactile physical selection oradjustment means (such as knobs) common to dashboards today which may bealso operated in conjunction with the surface distortion or other typesscreens disclosed herein. What makes it novel and exceedingly useful isthat it has programmable visual and tactile aspects approaching and insome cases exceeding today's dashboards, while at the same time beingprogrammably changeable as needed to provide added features and enhancedversatility. By doing so, safety is improved, and user value is enhance.

Elements have been disclosed in the referenced co-pending applications.Additional details are provided in FIG. 2 and other figures.

As shown in figure, an instrument panel center stack 101 is equippedwith a large panel 105 which is primarily a display screen havingscattering characteristics to act as a rear projection screen for LCD,DLP, or other type of computer display projector 110 positioned behindit when viewed by driver 111 from the drivers side, and controlled bycomputer 120.

Clearly by known means computer data including messages and othercommunications down loaded to the computer from external sources, or thevehicle itself can be projected on the screen.

In this embodiment however, knob 115 (and if desired other knobs ortactile physical selection or adjustment means such as 116) is mountedon the screen so as to be rotatable thereon to various positions whichare sensed (for example by optical means as shown in FIG. 2, such ascamera 117 which looks at points on the back of the knobs which relateto their circumferential position) and reported to the computer 120which in turn indicates the knob position, functions or other data byprojecting same onto the screen. The rotation can optionally have adetent function, or another feeling sensation dependent on rotationalposition can be alternatively provided by an acoustic source such as 125programmed by computer 120 and driven by drive electronics 127 which, oncommand sends acoustic waves 126 which can be felt by the user thru thescreen panel 105, which is made of Plexiglas or lexan.

By sharing the display area between knob selection or adjustmentfunctions (and their description/instructions, and display functions(e.g. display of navigational charts), space is saved on the dashboard,and larger knobs and lettering may be provided (especially given theprogrammable operation). This then promotes safety immediately by makingit easier to see what is desired, or has been effected. The screen toocan be a touch screen as well.

Contributing even more to safety, the tactile feel of the knob, or othertactile physical selection or adjustment means, can itself beprogrammable, for example using programmable acoustic wave pulses orother means, giving many added benefits. Note that such a programmabletactile response can be programmed to change with function selected,and/or variable affected And can operate statically too, to give thedriver a chance to tell the setting of the knob by feel alone. Inaddition, conventional cues to the driver such as the displayed valuesor computer generated speech can be used as well or instead.

While described as a knob movable rotationally, other tactile physicalselection or adjustment means like sliders, switches, levers or the likewhich are movable linearly, angularly, or in other manners can be used.

Let us consider in more detail the operation of the preferredembodiment, also in consideration of the front view of FIG. 1 e. Asshown, there are two knobs, 115 and 116 preferably large for ease ofuse. While illustrated vertically, a horizontal or any other positioningof such knobs or other tactile selection and adjustment means can beused.

A driver can look at this screen 105 and see the lettering displayed inbig letters next to knob 115 indicative of its function, eg the word“HEAT” as shown. If he needs to keep his eyes on the road he can as hetouches the knob receive a physical sensation due to the acoustic source125, which typically is a piezo-electric transducer known in the art.For example, if as he touches the knob 115 the computer 120 can causethe driver to feel a series of pulses 1 sec apart which indicate thatthe knob is a function selector knob in its current state of programmedoperation. Or the one hertz pulses might mean that it was the heaterfunction (if there was no doubt as to what the knob was for—ie having asign with big lettering right next to it.) or perhaps it was permanentlysuch a knob.

Alternatively, the definition of the setting can also be actuated byvoice. Say status, and the HEAT indication is displayed.

In any case as the knob is moved, the lettering projected is changed toindicate the new position. For example in one program, as the knob isturned 20 degrees rotationally, the function changes to wipers fromheat, and in so doing the acoustic source causes 2 hertz pulses, (oranother choice of signal like alternating high low pulses etc) whichsignify the wiper function.

The system is totally programmable—the degrees of turn to create a newchoice of selection or function, the acoustic pulse choices, and thevisual display of lettering (and language thereof too) is allprogrammable in computer control unit 120. If desired a computergenerated sound (which can be made to emanate from one or more of theradio loudspeakers can also be used to indicate position as well orinstead of the acoustic wave tactile pulses (or other types).

This system has the ability to have its data read in a quick visualglance and/or touch and visual confirmation too after a move is made.Voice input or output can also be used. For example when the knob isstopped from being turned, the computer can annunciate its position. Orthe person can tell the system via a microphone connected to thecomputer equipped with a voice recognition program (not shown) what hewants the knob to be (eg heating and air) and the knob function and itsassociated display can be changed accordingly.

When the knob is at the point desired in the circumferential direction,it can be left there and after a momentary dwell, the computer registersthe reading desired (e.g. wipers at the second position). Or, in anotherexemplary mode of operation, the knob function can be changed. Forexample, when at the “wipers” position, the knob can be pushed in toregister this choice (wipers) and then after that the knob function andthe display associated with the knob changed to wiper speed delay andother wiper function settings at the different circumferential positionsdesired.

Note that the unit may have different functions made possible for theknob or other devices when the car is stopped. For example the knobmight indicate wipers, heat and radio when in motion, but additionallywhen stopped could have email, internet surfing and other functionswhich might be too dangerous while in motion.

As an alternative to an acoustic source thru the screen to signal a feelof a variable to the user, a sound generator loudspeaker 130 can sendsub audible waves 131 against the whole screen 105 which can be sensedat the point touched.

In the preferred embodiment shown, the portion 106 of the screen 105 isreserved for touch screen inputs for example using known resistive,capacitive or other means such as my distortion based types. Similarlyas shown, the users finger 141 (FIG. 1 d) touching screen 105 can inputdata, and information can also be communicated via acoustic source 125controlled by computer 120. (see also FIG. 14).

It is noted too that the touch screen can desirably have indents orridges such as circular depression 140 in screen 105, in order to guidethe users finger such as 141 to a certain location on the screen wherefor example the starting point of various command movements might bemade. The indent or ridge (disclosed further in FIG. 5 and elsewhereherein) can be shallow such that it can be felt, but not deep enough tocause refractive gradients which would disturb an image say of a mapthat might take up the whole screen surface in a navigational mode forexample. Ridges and depressions in smooth surfaces of even 0.003 inchesfor example can work (thickness of human hair), while even 0.020 inchesdeep or high can be used with little optical effect in many cases.

It should be noted that because the display and tactile physicalselection or adjustment means providing control functions and feedbackare both programmable, one can have programs which vary by driver.Indeed, one can even take your program with you, for example If rentinga car having a similar display and a data input device for your to enteryour program in (e.g. a CD Rom for example using a version of the CDplayer of the car). Or your program can be downloaded from remotesources such as your home computer or the internet (where a selection ofprograms might be found, say on the GM web site).

The Sequence of actions undertaken then using the embodiment in onepreferred version is 1. Glance, 2. touch, 3. move, 4. confirm (tactilyand/or visually—much like today's dash boards—but generally with evenmore tactile feel relating to the position, and larger lettering whichcan be better seen at a glance) Alternatively, one can do it entirely byfeel, using the techniques described above where the programmableacoustic source is used to input to the user all data needed (even toinclude the initial starting point of the knob position such as a shortpulse burst, with a long delay until the next one to indicate a firstposition, or the end point, which might be three short pulses with along delay, to indicate position 3 for example. Other pulse frequenciesor codes could signify different programmed knob functions if desired,like wipers or heat, or whatever. Or voice can be used as discussed, orcombinations thereof.

Also gestures can be used, such as hand or finger position or movement,as disclosed in my copending applications for example.

It should also be noted that using the computer controlled projector (orother screen) display a variety of visual cues can be used to signal afunction or state has been reached. For example, Not only can onedisplay a word such as “high” to indicate high heat, but one can alsohave it blink 3 times when reached. This could correspond to a acousticor other tactile signal comprising three pulses as well.

Also it is possible for the colors or patterns of the words or figuresto be changed programmably. For example, in the FIG. 1 case the wholeknob, or its surroundings could be illuminated through use of anappropriate computer program for the display, in bright red when thehighest heat position was chosen (with blue, for the lowest, forexample). And for example, the knob surroundings could be projected onthe display in polka-dots, if the knob was not in a position thatactuated a function (e.g. a dead zone).

Where desired (e.g. with elderly drivers) the writing near the knob,might be in large letters, which could even be so large that the words(such as “high”, for a heater blower speed selected, would need to beabbreviated as HI” for example).

It should be noted that the knobs or other tactile input devices showncould be contained on an interchangeable member. One can have a touchscreen, equipped with interchangeable front panels of the type justdescribed which have specialized physical devices for interaction withthe sensing arrangement used. In this manner one can actually add in aninterchangeable manner, overlays to your dash which can be used fordifferent purposes. This can enable you to do tasks by feel nototherwise possible.

For example consider a tactile display or touch screen of the type shownin FIG. 1 or 2 using optically sensed details on the back of the variouscontrol items used. In this instance the control items are able to beinterchanged. A group of levers could be on one screen, knobs onanother, and so forth. Or one screen could have markings on its face foremail and stocks, with levers and knobs to suit, etc.

This overlay type of arrangement does not depend only on the touchscreen types I have invented, but may work with touch screens employingother technology, for example the acoustic surface wave version of FIG.12.

The tactile control of the computer system used can in this manner, becustomized not only by the user in general, but for specificpurposes—what I call a “Mission specific” tactile and visual input andoutput. This makes it still safer by providing just the right tactilecontrols for the application at hand.

To illustrate, consider optional removable (and interchangeable) tactilescreen member 144 (dotted lines), in this case a piece of plastic shapedlike a 3.times.5 inch card, and containing a knob 145 (also dottedlines) as described whose position can be determined (optically viacamera 117, or otherwise).

This card member can be inserted to change the function of the tactileportions, or for example to include, as illustrated in this case, anadditional knob as tactile physical selection or adjustment means. Inone preferred embodiment, the card member can be placed in slot betweenguides 146 and 147 (which could be dovetailed to assist). Dataconcerning the function of the knob in this particular case can beprovided to computer 120 by any means desired, including downloads fromthe internet, magnetic or optically encoded data on the card orwhatever.

The card as well can have printing on it, and may be transparent inareas desired to allow data from the display behind it to indicate datarelating to the knob position or other variables. The use of datapermanently printed on the card might befit a mission specific or onetime use card, for a particular project, or a particular drive to work.Note that the card could also correspond to just one function, like aninternet music source selection card, that you could use just for thispurpose, removing and storing it when you wanted to put another card in,or free up the display area for other information.

It should be noted that an acoustic source such as 125 providingprogrammable feel, can couple to this insert-able member as well. Suchcards have a very interesting use as single use, or mission specificcontrols for certain purposes, and provide additional freedom in choiceof tactile physical selection and adjustment means.

It is noted that TV camera based systems are not the only way ofdetermining position or orientation of tactile physical devices such asknobs in the invention. Other technologies may be used. For example ofinterest in rear projection display based applications of the inventionare DLP or micro-mirror type chip projectors which can be used in areverse mode as a scanning device, capable of determining, with asuitable detector in conjunction therewith, indications relating to saythe location of a flag on a knob, corresponding to its rotationalposition. In some cases they can be used to see finger touch as well, byfor example, the direct viewing method of FIG. 11.

FIG. 1 f provides another illustration of such devices. Consider aversion, in which a transparent member 150, typically glass or Lexanplastic is placed in front of an LCD or other flat panel display (e.g. aplasma display) 155 having front surface 156 and rear illuminationsource 157 which directs light upward in the drawing. LCD displaysoperate by switching bi-refringent properties of individual elementssuch as 158 160 (of hundreds of thousands), whose size is exaggeratedfor clarity.

Knob 165 is secured to transparent member 150 using pin 166. On the rearof the knob, is reflector 170 which reflects light 174 from light source157 passing thru the LCD elements such that light so reflected (or otherwise deviated) back into member 150 reaches high gain photo detector 175by bouncing between the front and rear surfaces 176 and 177 of member150. The reflector design can be such that it scatters or otherwisehelps direct light to the detector.

In operation, the location of reflector 170 is determined, in order todetermine the circumferential location of knob 165. In this example, thearea of LCD pixel elements in the region of the knob are all initiallydark (in this case a circular area), and one by one are sequentiallyopened up (under computer control) to allow light from source 157 topass thru in order to illuminate the region in front of them. Toillustrate, with the knob pointed in the position to the right of thedrawing, as the pixel elements 158 160 are sequenced thru, it is noticedon graph 180 of detector amplifier 181 output, that at as element 159was illuminated, that detector 175 received a signal considerablystronger than at other positions. This then indicates that the reflectorwas closest to that represented by LCD element 159. Typically such aposition would be an grouping of elements able to identify knobcircumferential position within the tolerances needed (e.g. every 10degrees, say).

LCD elements can be very rapidly sequenced in many cases. Thus it may bethat a sequential illumination scan such as just represented can be doneso fast (e.g. in a millisecond) that it is unnoticed by a user—even ifhe is looking directly at the knob. (human eye integration time being 60milliseconds.). To make the total scan even quicker, it is noted that itis only necessary to scan an annulus corresponding to expected reflectorpositions, if a point reflector is used on the periphery of the knob,say. Other reflector designs can also be used. And refractive,diffractive, or other elements to direct light can also be used.

It is noted that transparent member 150 may be interchangeable as well,as disclosed above.

As shown therefore the circumferential location of knob 165 has beendetermined. Similarly slider (or other tactile selection or adjustmentmeans) axial locations can be determined as well. And in addition, socan finger touch locations in x and y dimension of the LCD panelsurface.

For example consider finger 190 touching front surface 177 of member 150as shown. In this case the finger itself acts to reflect light 191 fromindividually sequenced LCD pixels (not shown for clarity) to thedetector and its location can be determined by similar means.

Finally it is of interest to consider optional ridge or other smallprotrusion 192 on LCD flat panel screen (or cover glass thereon, whichas noted can be interchangable). This as noted is to provide a tactilerelation between the users finger and the screen and display, an issuediscussed in FIG. 5. This allows the finger to easily find a set pointor line of action on the screen without looking. Typically the height ofthe ridge or alternatively an indentation is small, so as not to undulydisturb the light field of the screen. Generally preferable is whenthere is sharp slopes, so as to minimize lens effects.

It should be noted that displayed data next to (or even within same, ifroom exists a knob or other tactile physical selection or adjustmentmeans usually at a minimum displays the selection or adjustmentchoices—e.g. wipers, heat, air, cruise; or lo, med, hi for example.Alternatively or in addition, pictographs, colors or patterns may beused for example.

In addition other information can also be displayed, such as more detailabout what the choice means. This can even include effectivelydisplaying the instruction manual for that function if desired.

FIG. 2

FIG. 2 illustrates in further detail the versions and features of thepreferred embodiment for near term commercialization disclosed inFIG. 1. It is illustrated in the form of a rear projection display, butis not limited thereto. This embodiment senses positions of screenmounted knobs and other conventional tactile selection and adjustmentmeans. Such sensing can be performed by a variety of electrical,magnetic, acoustic or other means known in the art, but here ispreferably illustrated with optically based sensing means, which has theadvantage of simplicity and non contact operation.

Consider screen 200, where a physical tactile selection or adjustmentmeans such as a knob, (switch, slider, rocker, etc), 203, is in oneportion of the screen surface, and this knob has on its rear facing thecamera a “pointer” or mark 210 which can be imaged and sensed by thecamera 205 and its associated image processing computer 212 to tell theangular knob position, and pass this information to the display andvehicle control computer 215. Camera 205, can also be used to see screendistortion under touch like 340 in FIG. 3. This means that a combinationtouch screen and “regular” function dash can be provided for very lowcost, which can be interchanged, since the camera 205 can be programmedto see any effect in any zone. Note that the light source of projector218 even may be used to provide illumination of mark 210. And since onecan control the projector, one can choose the light projected forexample to clearly illuminate the marker—which itself may bepreferentially reflective of a certain color which may be instantlyrecognized in the color image obtained by camera 205, which can be asolid state matrix TV camera. Identification of the marker is also madeeasier by the fact that its relative position is approximately known tobe in a certain region of the screen.

While the projector source may be used to light the markers of any knobslider, switch or other selection or adjustment means, a separate lightsource such as laser light source 219 can be used alternatively suchthat the camera using filter 221 (dotted lines) can for example see,using an interference filtered image responsive only to laser light, theknob mark reflection (where it too can have a dichroic mirror reflectingonly laser light substantially). This makes the sensing of knob positionindependent of what is being projected, and it can operate with noprojection at all. Its noted that if the laser (or alternatively forexample an LED source) is in the near IR (e.g. 0.9 microns) the userwill not see any indication of this.

With proper construction, the knob or for example transparent slider230, having mark 231 (also viewable by camera 205, or a different cameraif desired) running in track 235 in transparent screen 200, can be alltransparent (however, scattering light from the screen face, 201. Undercomputer control, the projector source 218 then can be used to changethese devices to indicated different functions, both by projectedlettering and color, and possibly other patterns or the like.

For example consider the function of transparent slider 230, illuminatedby light projected through it, 231. as shown this runs in dove tailgroove track 235 in the outer portion of screen 200, as shown in FIG. 2b. in one mode, shown in FIG. 2 c, the word “HEAT” is projected on thescreen over the top of it, by projector 218, and further, the zone froma to b in the slider grove may be projected with a continuous bandchanging from red (hot) at point a, to blue (cold) at point b. Camera205 senses the position of mark 231 and computer controls the heatingand air-conditioning system of the vehicle accordingly.

In another screen scenario however, the projector can turn this sameslider into another function altogether, for example as shown in FIG. 2d, where the data projected is indicative of the viewing direction of aswiveling exterior rear view camera on the right side of the vehicle,which presents an image 240 of the rearward TV camera scene viewed onthe same screen 200 (or alternatively, a different screen), by virtue ofthe projector 218 also projecting the image. The words right and leftare projected in this case.

In this manner, a relatively small number of knobs, sliders, switches,buttons, or whatever can be used to control a large number of functions,while in large measure still retaining a classical “feel” to which usersof dashboards are accustomed. Not all portions of the screen zone needbe transparent, allowing the projected image to be displayed at allpoints, but this feature allows maximum flexibility in the use of thescreen. The surface distortion technique or other touch screentechnologies as desired, may be combined with the direct viewingtechnique as can be appreciated, such that more data can be entered aswell. For example in a particular combination, one can use the FIG. 2version to dial in a heat setting, and then push the knob in, creating adistortion type reading as taught in FIGS. 2 4 above and co-pendingapplications indicating that the selection has been made. Or one can usestereo vision or other means to detector the push command—which also maybe proportionally related, rather than just a switch.

Its noted that not only then is the image displayed programmable, butthe inputs are too. for example, if problems were detected, theMalfunction screen (or an applicable portion thereof, such as electricalsystem data, if an electrical problem was detected) could be called tobe displayed, and the slider 230 would assume say, the function ofturning the hazard lights on, and calling “ONSTAR”—if not automaticallycalled.

It is noted that the knob and slider examples, are those where theposition of the device (angular or linear, in the case of knobs andsliders) is determined, and provides an analog or digital indicator (ifconverted, or if mechanical detents area used) to the control computer.If switches are used, the function can be as simple as on or off. Inthis case, shown in FIG. 2 e, the button, such as 260, might push in tothe screen 265 to turn on, exposing as it does so a mark 270 to camera275 (like 205).

It is useful to have tactile feel provided with ratchet clicks forexample, when moving a knob or slider or other means of the invention.This provides a feedback to the user of how much movement has occurred.In addition, the system itself can provide a feedback by sound, or byvisual indication on the screen which could be a change of pattern,color or the like.

FIG. 2 f illustrates another example of a tactile conventional mechanismbuilt into a screen—which itself may optionally be a touch screen. A bigknob 277 as above is for example located in the middle of a large screen278 in the central dash portion. Both the knob and the lettering on thescreen are desirably large to allow most all drivers to read it easilyin a glance without the aid of glasses. Optionally the print and controlknob etc sizes could be user selectable. For example a young user mightlike four small knobs rather than one big one, with less need to go tosecondary screens. Another user might want sliders rather than knobs.With an open computer system, programs for this could even be purchasedin the after market.

As shown there are 6 indications at different circumferential locations(as shown there are diversely illustrated as Heat/Air, Wipers, Stocks,Email, audio. It is possible to have more than 6, with each still beingquite readable, especially if the screen is large.

The user turns the knob to the desired indication and presses in theknob as appropriate to indicate his selection, which is detected by thetouch screen apparatus or by a second camera capable of seeing the knobfrom a different angle and thus by matching with image from camera 205determining depth movement of the knob by pressing. (see other drawingsfor examples).

Then as shown in FIG. 2 g, a new screen 285 (in this case a heat oneselected in the first case 2 f for example) may be generated by thedisplay control, indicative of that function. In this particular examplea bar indicator 286 is displayed, and the user touches the point “P” onthe bar which he wants, temperature wise that is. The point can beprogrammed by the display computer to be color indicated, or with actualtemperature markings or whatever. Or he can use the knob to select a fanspeed, rear air distribution, or something else.

FIG. 2 h illustrates a version of FIG. 2 f or g which can be achievedwith a conventional display screen having no special capability at all.In this case a conventional knob 290 (or other tactile selection andadjustment means) is located below screen 291 and a large displayedsurrogate knob 292, like 277, is displayed on the screen. As the userphysically turns real knob 290, the surrogate knob 292 moves to thepositions indicated—just like in FIG. 2 f, where it was physicallyturned. Again, one can push the real knob in if desired to indicate achoice. Or the choice can be indicated by just leaving the knob inposition for a minimum time period.

The “real” knob of 2 f (albeit virtually displayed), or the surrogate of2 h, both allow displays having multiple choices of different variablesor functions, and different gradations at different circumferentiallocations, and the use of different languages or the like.

It should also be noted that such a conventional approach can be stillmore conventional, and that is to not display the surrogate knob 292,but rather just display the status of the knob and/or the result of itsmovement or other action.

For example, consider display of the word “Heat” 295 in big letters onthe screen, as the knob 290 is touched and movement is sensed. As itreaches its next setting which could be “Wipers”, for example, thedisplay would be changed accordingly. These large words could besupplemental to any smaller display words chosen. (particularly when theknob is on the display rather than below it or otherwise positioned).

It is noted that the acoustic or other programmable tactile feedback tothe user operating the conventional knob 290 can be provided as well.

It is also noted that with TV camera sensing of a finger touch screendistortion or features of physical tactile devices such as knobs asdescribed can be usefully accomplished by change detection, accomplishedfor example by subtracting images in a quiet previous state (i.e.static) from instant images. Both finger touch and knob turningrepresent change which can be so identified. And the region of changepinpointed.

Its also noted that one can calibrate the camera system each time ameasurement is made as well, if the movement is known—e.g. a knob turnto the next position

FIG. 3

Where surface distortion based transduction of touch events is desired,a preferred means of detecting surface distortion is that given inreference 1, which discloses illumination of a surface and subsequentretroreflective re-illumination of the surface from which an enhancedimage of the distortion in such surface are created. This method (andthe products based thereon sold under the trade name “D-Sight™”), is atonce, simple, fast, and capable of intelligibly measuring minutedistortions over large surface areas. All of these are advantages forthe present disclosed invention, and “D-Sight™” is the preferred method(but not the only method) for determining such distortions. Otheroptical techniques are grid and moire triangulation, also providingsurface distortion data.

FIG. 3 is a basic embodiment of one type of distortion/deflection basedtouch screen suitable of the invention, previously disclosed inreference 2, FIG. 2 (and others). Touch screen 301 located on a vehicledashboard 302 in front of a passenger 305 for example, typically anddesirably of significant size so as to make reading and touching easy,operates as follows. The screen surface 301 is illuminated by a rearprojection TV projector 310, in this case provided by a low costmicro-display such as an LCD matrix 311 or alternatively for example, aTexas instruments DLP device, illuminated by light source 309, andimaged on screen 301 by lens 313, and controlled by display computer315.

A TV camera based transduction of the screen touch or other physicalfunction is used as described in co-pending applications and further inthis application. In this embodiment, surface distortion of the screencaused by touch or impact of one or more objects (e.g. thumb 320 andfirst finger 321 of the passenger) is a achieved using for example the Dsight effect, and employing TV camera 340, light source 345, andretro-reflective screen 350, together with microcomputer 360 (which maybe the same as 315) to process the data obtained as to the location, ofany and all contacts in the xy plane of the screen, and their zdirection or force function if desired. As was disclosed in thereferenced application, it is the local screen distortion caused by thetouch that is being sensed in this case.

It is noted that camera 340 imaging distortion of screen 301 touched byfinger 321 may also be used to image a projected image on the screenproduced by image projector 310 and backscattered by the screen.Analysis by computer 360 attached to camera 340 directly correlates thepoint of touch “P” to a point determined by 360 in the image. Forexample If a test image is produced on the screen which has 100 verticalbars, and the finger touch hits between the 90.sup.th and 91.sup.st bar,it is then known that the reading of the touch determining apparatuscorresponds to this location in the image. Similarly 100 vertical barscould be projected and the vertical distance calibrated.

This correlation can be performed when ever it is desired to provide atest pattern. Thus the operation of the device is not dependent onstable image projection over time and a knowledge from the projectioninput, as to where the touch occurred. This kind of operation is usefulif there is instability in the projection for example, or if there areproblems with the mounting of projection devices, large amounts ofthermal expansion or the like.

The transparent screen material may be darkened if desired, to make thesystem innocuous and or invisible when not turned on. If desired, thedisplay may be flat, or curved with reasonable positive or negativeradius to correspond to aesthetic design considerations. (generallyhowever, flat displays are easiest to use).

While not the only form of touch screen which can be used with theinvention herein, this distortion based system has several uniqueproperties which are helpful in the car application. For example some ofthose given in reference 2 are: A potential “four” and “fivedimensional” capability, wherein the force vector direction as well asthe magnitude of force is measured. An ability to have a data storage ofa complete signature at once, physically or in memory. Robust andreliable. Many prior art touch screens are of specialized constructionand would be quite expensive to replace if they were broken, especiallyas the size increases An ability to have the surface distortion ortouching input means of any material, completely removed from the actualsensing of the input. Inexpensive, particularly for larger surfaces.Some embodiments give a desirable tactile feedback since it is theactual physical deformation (and the amount thereof) that is responsive.Thus the feedback to a finger (or other member) in terms of resistiveforce is proportional to the desired input. This tactile feedback isparticularly desirable in for example the automobile where one shouldnot take one's eyes off the road. High resolution data entry may be madedirectly on the screen with ordinary instruments or fingers. If desiredthe “beam” of the display can literally follow the touch point, just asa pencil line would follow a pencil. In this application the 3-Dcapability allows one to press harder and make a darker (or wider) linefor example, just as one would do in normal practice. MultipointOperation. The existing touch screen art is capable only of measuringone touch point in X and Y at a time. The invention is however, capableof multi-point operation or even simultaneous detection of complex area“signatures”, not just “points”. A further advantage of the invention'sability to detect multiple input signatures, etc. at any point on itsface, therefore a keyboard, a piano keyboard, a joy stick can beartificially created at any point under computer control or simply byrandom human command. This is particularly desirable in a car where youcannot necessarily keep your eye on the data entry device. And it is ofuse for handicapped people who could not be expected to hit the rightpoint on the device every time, but if they just hit the deviceanywhere, could make a move from that point in a manner that would beintelligible to a computer for example Variable and “Intelligent”orientation. It is also useful therefore to replace in many caseskeyboards which have continuous arrays of keys, be they optical,mechanical, contact, electro mechanical or whatever. Unlike mostkeyboards the disclosed type can “float” (i.e. be at any zone on thesurface) which is convenient for people who know how to type but cannotsee the keys for example, while driving. Tactile feedback, includingprogrammable. The unique tactile feedback application aspect of theinvention allows one to essentially use a deformable member as a sort ofminiature joy stick for each finger or to allow one to rock back andforth between one or more fingers to enter information, for example—suchas seek controls on a car radio. In addition, programmable tactilefeedback such as air blasts, vibration, etc., can also be added easilyto the touch surface.

Another advantage of the invention is that it can detect a force ordisplacement signature such that the signature of someone can be used asa control enabler, or other verification tool—for example in authorizingthe driver to start the car, or an internet transaction. In addition,for the handicapped, non-conventional inputs such as palms and the likecan be used, rather than fingers.

As was pointed out in copending applications, the distortion basedscreen, because it physically is perturbed by the finger(s) of the user,has some subtle tactile features all by itself. For example considerfinger 320 and thumb 321 of person 305 who desires to “rotate” a virtualknob 365 (dotted lines in FIG. 3 c) depicted on touch screen 301 (forexample created by display computer 315). The person can do so bypressing in on the screen 301, and registering to the computer that afinger 320 and thumb 321 are touching in close proximity, just as onewould pinch a small knob on a dash of today. This is illustrated in FIG.3 b. Then in a sequential motion, the screen sensing system senses thatthis knob is being turned an amount m, in direction theta, so to speak,as the fingers rotate their points of contact indication, FIG. 3c—effectively like a twisting motion. This amount of rotation iscommunicated to the car control system, and the heater output let ussay, raised accordingly.

In another example shown in FIG. 3 d, an input of how much windshieldwiper delay is desired, is achieved by just touching the screen 301again in this case with finger and thumb, spaced d apart, in the generalarea of windshield wiper control functions (dotted lines for example).This could be a fixed zone of a particular displayed screen (such thatthe driver would know by heart, or could be projected on the screen forthe moment in such large manner that it would be easy to see—and hit, soto speak. (so one wouldn't have to take eyes off road for but a fractionof a second). With fingers still resting on the screen, the driver forexample, could move them a distance delta d to a new delay position, inorder to decrease the delay, say.

Alternatively the distance w of a finger touch point 338 with respect tothe fixed base point ‘a’ (eg at the side of the screen, or from a ridgeor other tactile reference) could give the delay for example.

Consider too the action of finger 320 and thumb 321 which can both reston the screen at once, and alternately push in and out, so as to “rock”back and forth, just as one might do today in selecting radio stationswith a seek rocker switch, or moving outside mirror positions forexample. While use of a finger and thumb are most natural, clearly anytwo things could do it in the above example. And some functions mightuse three or even more input points.

More common is just one input, such as a forefinger. Here again thedevice of ref 2 is useful, as the forefinger force on the screen (also az input) can be the function desired.

For example, as shown in FIG. 3E, a box 370 can be displayed on screen301, and the person just pushes on the box at point ‘P’ with a forcesufficient to cause an indentation or distortion ‘Z ’ which is sensed,and used to indicate that a high heater blower speed is desired (the Zvalue attained could also be indicated to the driver with a tone orother audio pulse, or a visual indication on the dash. For example avalue of 0.010 inches could give a louder or higher pitched sound than a005 inch movement. Or it could be fed to a voice program to just sayHIGH, for example). This is easier than, and preferable to, the commontouch screen alternative, of leaving ones finger on the box for a givenamount of time t, during which the heater speed would ramp up. Thelatter is disadvantageous, in that one must wait—with ones finger stillin place—for the desired speed or other desired result to be obtained.

It should also noted that it is thought possible that one can push in ona surface distortion detected screen with one object, such as aforefinger, and not only detect the degree of indentation, but thedegree with which one might “twist” ones finger in a rotation motion.This makes it easy for a driver to interact.

It is also noted that one can also write on the display screen 301 withany object for example pen 375 and have the computer trace the line suchas 376 on the screen, or store it memory, or transmit it (the same goesfor all data inputted to the system by whatever means). One can alsosense the head 380 of pen 375 with a camera pair 385 and 386 (shown herein the windshield header) such that the point of contact is known on thescreen, and the head position is known, and thus the pointing angle tothe screen is known which can be useful for various applications, suchas pointing at data within a 3D virtual display.

Note that as alternative to the twisting knob motion of FIG. 3 b cabove, one can also just trace with ones finger, a movement on thescreen. For example, a small left to right linear movement starting atany point in at least a region of the screen, could indicate an increasein a variable such as heat, where as an arc shaped movement in theclockwise direction could indicate an increase in heater blower speed.The more the movement, the more the increase (and like wise decrease).The rate of movement of ones finger across the screen can also be sensedby comparing consecutive camera views, and one could use rate to give avalue as well—a quick movement could be a quick blower speed, or a fastwiper speed, say

FIG. 4

FIG. 4 illustrates the same arrangement as FIG. 3, but with the imageprojection device 410 located off axis at angle alpha to the normal 415of screen 430 as shown, such at an airbag 440 can be deployed on axis,substantially along the normal. The display can be a simple displayonly, or can be part of a touch screen as disclosed above and elsewhereherein.

The airbag when initiated moves the touch screen 430 out of the way. Inthis case it is desirable to secure the screen in such a way for exampleby providing it with a weakened tear line, that it blows down, or out ofthe way, and to make the screen of material which itself causes minimalharm. The ability to choose the screen material accordingly is advantagewhen using the screen distortion based touch sensing technologydisclosed in ref 2. However, other touch screen techniques known in theart can be used. For example a surface acoustic wave generator 441 canbe used (with acoustic microphone pickups not shown for clarity), totriangulate on where the human touch 443 has modified the acoustic wavepassing through the screen material.

Both optical and computer correction techniques are used to make apresentable display. For example, the projection source may have a lightsource 450 illuminating a LCD matrix 451 driven by display computer 455,where the matrix is tilted at an angle beta with respect to the axis ofprojection 460 such that the image is in focus at all points on screen430.

Since this arrangement yields varying magnification at different pointsacross the screen, the input to display computer 455 is advantageouslycorrected using equations processed by said computer to account for thevariation in magnification and other issues as desired, in order to makethe projected image appear uniform.

As disclosed copending applications, the point 479 of an ordinary pen orother device held and manipulated by a person is shown here as an input,for drawing or other purposes using computer 485 connected to theaccoustic sensing system 441, or alternatively, electro-optically basedsensing system 442. This computer 485 also acts as input to displaycomputer 455.

Finally it should be noted that another advantage of the surfacedistortion system is that the material can be anything sufficientlytransparent that sufficiently deflects. If it is desired that the screenbreak apart, rather than fly off, swing down etc, when the airbagdeploys against it, it can be slit or serrated, or for example screen430 can be equipped with a weakened tear seam 491. Other touch screentypes may also be advantageously slit or serrated or otherwise inducedto break or disintegrate as well, but many will not work properly inthis mode due to disruption of acoustic or capacitive fields forexample. Again a big advantage of the instant invention.

This approach not only allows the airbag to deploy properly independentof the presence of the touch screen, but as well allows low cost ofrepair after a deployment. The screen surface isn't expensive, and isthe only thing struck-less cost possibly than repairing dash upholstery.

It is noted in FIG. 4 that the image projector can be angled in the x zplane as opposed to the yz plane as shown, in order to miss the airbagdeployment. Any touch screen sensing using cameras or whatever does nothave to be in the same plane with the projection unit. In addition,where it makes sense, the airbag itself can be somewhat angled withrespect to the touch screen as well.

FIG. 5

On a touch screen display, it is desirable to have tactile indication ofwhere to touch, ideally so one would not have to take ones eyes off theroad. FIG. 5 illustrates an example using a screen wherein such a meansfor tactile relation comprises a local disturbance or relief of thescreen surface.

First let us consider a touch screen 500 which could be of any type, butis particularly easy to construct using the surface distortion type. Thescreen surface 501 is not flat in this case, but has transparent localprotrusions such as 505, or indentations such as 506 which make findingthe zone near them on the screen possible by feel. Generally, it ispreferable that the protrusions or indentations be small, on the orderof 0.010 inches or less for example (with correspondingly small slopesif a smooth indentation), such that minimum discontinuity to the eyeoccurs when images are on the screen.

For example, round indentation 506, could be somewhat larger in diameter“d” than ones finger tip 507, and be a position where z force would givethe desired input—what ever it was. Thus if the driver switches screensto driver input functions, and his finger went over to touch theindentation in the approximate zone he knew, he would know he wastouching heater speed, for example. If his finger went to theindentation on the other side 516, he would know that was wiper speed,and so forth. He would not each time have to look at the screen.

Similarly, if his finger found ridge 505, which might be one of tworidges, the other being 515, he would know from the displayedinformation “Heat” the bottom ridge 515 was for example heattemperature, and he would slide his finger along the ridge in the xdirection accordingly to indicate how much heat was desired, forexample. If he wanted wiper delay, he could slide along the upper ridge505. In either case, all functions once learned, (ideally possible by avery quick observation if one was driving a rental car for example)could be found by feel.

With a distortion based screen, such indentations also serve anotherpurpose, in that they cause more local distortion in the zone indentedsuch as in 506 side view FIG. 5 b, due to reduction in wall thickness,t, of the material of the screen (e.g. Plexiglas or Lexan). This isoften useful to decrease the force required for a given resolution. Asimilar effect can be achieved as shown in FIG. 5 c by substitution ofmaterial of the screen, for example If in screen 540 a zone 545 wherecontact was to be made (in response for example to displayed functions)could be of material 546 different than that of the rest of the screen540. In this case the screen could still be flat, and if both materialswere equally transparent, likewise transparent. This is a particularlyinteresting thing, in that functions what ever they are could bememorized by the driver as always occurring in these spots on thescreen, even though the screen could be used to display anything. And insome cases the totality of the screen at all points could be touched,even though the response would be different to forces at differentpoints.

Indeed the previous indented or raised screen version, if the indentsdid not overly optically change the information displayed, can serve thesame purpose. This then leads to yet another alternative, discussed inreference 2, that is to provide an overlay, on an otherwise uniformscreen. This overlay can be of ridges, as just discussed for example, orcan actually include various real knobs and switches, which themselvescontact the screen. (previously disclosed in FIGS. 12 and 14 ofreference 2).

The idea of adding ridges or indentations to touch screens to improvetheir tactile function may be employed with other touch screens than mydistortion based type, such as those of the resistive type comprising amatrix of elements which contact at a certain point touched when theforce of touch presses the outer layer of contacts against the inner.But it may not be possible to easily provide such ridges andindentations on conventional touch screens of today employing capacitiveor SAW technology. It would however, be possible for indirect sensingmethods, for example the “touch screen” of FIG. 11 in which ones fingerimage is sensed using cameras or other external means, independent ofthe screen itself.

As shown in FIG. 5 d, the action of the indented other material 546 forexample, could “snap” to a new more indented location such that therewould be a definite feel passed back to the person touching it. Forexample consider button 555 which is curved toward the user in itsquiescent state, but when pushed in, snaps to position 560 (dottedlines).

Also of use is a screen version having zones of more elastic behavior,either due to weaker wall thickness of the screen, or different materialthan other parts of the screen. This may also be in addition to the useof indented or raised portions of the screen to facilitate the driverfinding the correct location for input action. Note that where desiredthe material may be of the substantially the same index of refraction asthe base material but considerably softer, and more easily deflected.And it can be flush with the screen so that the total screen appearsuniformly flat.

FIG. 5 e illustrates a novel annular groove 565 in a translucent flatscreen 570, which is easy for a user to under stand by feel, placing hisfinger 571 in the groove. A raised portion in the groove, 572, at the “9O'clock position” indicates the start of the groove, and as one movesclockwise, the amount of the desired input increases (typically, itcould be programmed otherwise). Optionally, small bumps (which forexample, can also be coded as shown, two 575 at 12 O'clock, three 576 at3 O'clock, and four 577 at 6 O'clock) can be used to further indicate tothe user the location of his finger input by feel. As pointed out above,the immediate region of the groove can be color displayed or the like tobetter communicate visually to the user the result of his action. Againthis groove need not be more than a few thousandths of inch deep inorder to be felt by the user. If the sidewalls of the groove are alignedwith a line from projector to the drivers eye, the effect of the grooveon the displayed image will be minimal.

The annular notch or groove is very “natural”. It has a neat feel, likethe dial of an old style telephone in a way, and the analogy to a clockis known by everybody.

Alternatively a circular or annular ridge sticking out from the screencan be made with some what the same effect.

As noted, a desirable aspect of such ridges, grooves and the like, isthat in linear form they can provide a “line of action”, so to speak,which the driver can trace his finger along, in the example from cold tohot, as one goes left to right. The ridge or groove can be under or ontop, or surrounded by the display as desired.

In FIG. 5 f, a more sophisticated situation is illustrated wherein aplot of stock prices for the day has been displayed (in this case in thesame region as the previous heat control, as a stock price selection hasbeen made), and the driver with a quick glance can tactily slide hisfinger 590 along a horizontal tactile ridge or groove 591 to the pointof the days trading (say 4 pm near the right of the chart) that he wantsmore information on. Such lines of action” can be vertical as well ashorizontal, and more than one can be on a screen (e.g. 592 in thevertical direction). And as disclosed above, the line can be on a screenoverlay which may be interchangeable, to accommodate different purposes.

Finally it is of interest to consider in FIG. 1 optional ridge 192 onLCD flat panel screen (or cover glass thereon, which as noted can beinterchangable). This as noted is to provide a tactile relation betweenthe users finger 190 and the screen and display. Of importance is thatthis is a very useful reference.

However to execute this with many types of conventional touch screens isdifficult. This is because one typically in the car dashboard case atleast would rest ones finger on the tactile ridge, indentation or whatever. This would cause a permanent signal if a conductive grid type, andthis may not be what is required. This can in one instance be solved byonly looking for changes in location of the finger touch, and whenchange occurs in finger position, then begin tracking the finger to itsfinal position—which final position then constitutes the signal desired.Alternatively, in some cases the path or other characteristic of themovement may constitute the answer.

FIG. 6

FIG. 6 illustrates a combined touch, gesture and voice activated systemWhich further illustrates the intimate connection of various functionsof the invention, further including voice activation and responsefunctions. The invention makes possible a multi-sensory dashboard datainput/output system, including voice recognition and computer text tospeech response, body part motion input (typically sensed optically, butnot necessarily) and touch screen capability. Clearly it would bedesirable to have such a device large, tactile, easy to read for allfunctions; and easy to add more functions—not achievable today withconventional dashboards.

FIG. 6 a illustrates a touch screen display 620 in which two TV cameras621 and 622 are located in the upper corners (or elsewhere) in order toobserve a users finger 625, for example in stereo to obtain 3D fingerposition if desired. In addition two microphones 628 and 629 are alsolocated so as to be spaced apart by a baseline b, allowing stereodiscrimination of a users voice location, as well as a voice command.The voice location is used to determine driver location, and optionallyfront seat passenger location for airbag deployment or other purposes.

In a first version, the finger 625 touches the screen and registers atouch location in x and y by whatever means (conventional or asdisclosed herein or in copending applications). The point touched by thefinger, lets say the box 635 labeled “Heat” is detected and thecoordinates of xy provided to the camera analysis module 640 by thetouch screen determination module 645. At this point, the camera systemthen knows the approximate location of the finger (or optionally part orwhole hand) in a very localized region of space, and can immediatelythen look for its indication.

For example, the finger can now be moved vertically up or down (or sideto side), to indicate more or less heat depending on the finger positionor movement. (heat having been the function selected with the touchscreen).

This is not the only indication of combined function. The user upontouch, can vocally speak the words “up” or “down”, with voicerecognition module 655 discerning his command and executing same.

It is also possible as another example, to do both, moving the finger upor down, and vocally indicating a command to the microphones and voicemodule 655. In this manner, if one of the signals is faint, or suspect,the other can prevail. Or a co-incidence type of logic can be used whereboth the finger and the voice have to agree, to execute the function.

Alternatively it would however be possible for finger location to besensed directly by external means such as the cameras 621 and 622,without any input from the touch screen sensing device.

FIG. 7

FIG. 7 illustrates a touch screen on the dash (either passenger ordrivers side), whose control is further (or alternatively) equipped, asdisclosed in co-pending application references, with an optional camerabased head tracker and/or finger or hand tracker (which can includemultiple cameras to improve accuracy and cover inputs from more than oneoccupant). As shown, in one example the camera 701 is located in theheadliner over the windshield 705 of the vehicle, in this casepositioned to look at the users face 710 (for face recognition securitypurposes for example), and/or to look at the users hands 711, fingers712, etc. Alternatively or in conjunction, camera(s), such as 720 can belocated in the dash 721 itself, which are also useful for augmentationof inputs of the touch screen by direct viewing of the fingers or handsof the user when near the screen.

A stereo pair of cameras such as 701 (and another located into the planeof the paper and not shown for clarity), located in the roof can be usedinstead of a single camera. The camera or cameras can alternatively belocated in the dash or elsewhere, and can also be used as taught in thecopending references to provide depth information of points in theirfield of view. Indeed cameras can be located in the dash to view theoccupants, or as shown in FIG. 7, can even be mounted in the touchscreen itself, looking outward from corners thereof.

For example, one mechanism is to use the upward movement of a personshand, such as 711, detected by camera 701 using image processor computer(eg a Cognex corp “insight” unit) 728, to provide as signal to thedisplay and screen input control computer 725 to change the displayeddata on the screen 726 by projector 730. One movement gesture of thehand (or just hand position) could be to change the screen from one typeto another, while another movement could indicate which screen. Or aspointed out in co-pending cases, the positions or movements couldthemselves indicate the action to be taken. For example If a sliderrunning left to right, corresponding cold, to hot of the heater wasdisplayed, an movement of ones hand or finger near the slider from leftto right could be used to provide an indication to the control computerto increase heat. In this case, a touch screen per se is not required,only a programmable screen. Thus this technique offers an alternative touse of specialized screens for touch input.

Features of the face, such as the eyes can be used too. For exampleconsider Camera 701 looking at the face of a person in the vehicle 710.In one embodiment, the persons eyes found in the image for example usingalgorithms such as discussed in U.S. Pat. No. 5,859,921 by Suzuki. Thepersons eyes are separated a distance d which is determined using thecamera computer system of the invention. As the face rotates, thedistance d shrinks (foreshortens) as seen by the camera, and thisreduction in d can be used to control either a function, like reducingthe value of heat, or it can be used to switch consecutively for examplefor every incremental value of d, to an new screen choice or actionchoice within a screen (e.g. heat action, vs. the same sliderrepresenting fan speed action, say). The user could, for example, evenchoose in one embodiment via a pre switch selection, whether he wantedto control such things by voice, by hand gesture, or by faceposition/rotation.

Indeed, the user can signal the system to change screens or some otherfunction, just by the number of fingers he holds up, the distancebetween them or some other signal.

As disclosed in referenced co-pending applications, by finding the eyesand nose, or eyes and chin, for example (e.g. 3 points), and using thelocations so determined, one can arrive a complete single cameraphotogrammetric solution (if the original spacings are known, or learnedby the system), such that x,y,z, roll pitch and yaw of the face relativeto the camera can be determined. The person can thus nod up and down tomake a selection as described above, using the camera to determine thevariation in position of the plane of the face determined by 3 pointssuch as the eyes and nose, or eyes and chin.

Another method to change the screens from one display to the next, or tochange the function of individual controls of a screen, is to provide inthe dash (or alternatively on the windshield header or door header forexample), a ranging sensor such as triangulation optical type 750located in dash 721, which looks outward in this case from the dash,and, as shown, measures the distance d, of a users hand 711 in front ofthe dash 721. Sensors of this type are described in U.S. Pat. Nos.5,362,970 5,880,459, 5,877,491, 5,734,172, 5,670,787 and others by theinventor and his colleagues.

This sensor can be set to read that a particular hand position has beenreached (as disclosed for example in U.S. Pat. No. 5,362,970), in whichcase the display for example can switch to a new screen—e.g. fromhighway screen 30 b, to traffic screen 30 c. Or in a more sophisticatedmode of operation, the sensor can determine the actual distance to theusers hand (or alternatively the change in distance plus or minus fromsome nominal mid point d0. For example, if the heater screen weretouched, the distance delta d from d0 in a positive direction couldindicate the increase in heat from the nominal setting. There areclearly many variations on how such controls can be structured, but allhave the advantage that they can operate in the car environment, and donot require one to deviate appreciably from important driving tasks. Itsnoted that with laser sources and solid state cameras (or PSD analogcameras) having large dynamic range, that operation in a car is assured.

It should also be noted that cameras such as 720 in the dash, can beused with laser or LED sources, such as 750 in the triangulation mode,and at other times can be used for photogrammetry or recognitionpurposes.

Other types of sensors can also be used to trigger screen changes, orother wise input data to the computer control systems of the invention,these can include, besides voice, other audio inputs (e.g. hand clap),proximity sensors and bulk mass sensors operating on inductive orcapacitance principles, and the like.

Sensors such as cameras or triangulation sensors such as can also beused as components of a “Smart” airbag control system to determine thecharacteristics and locations of occupants of the vehicle, as disclosedin reference 5. It should be noted that by co-locating camera or camerasor other sensors with the touch screen operation and control “module” tobe inserted into a dashboard, one can assure proper line up of cameraviewing and touch screen display, using a factory calibrated module,without in situ registration requirements (such that a indication say ofa finger pointing at a point on a screen, corresponds to the knownprojection of information at that point.

It is noted the dashboard which may have a platform for a lap topcomputer to dock into under the passenger side of the dash, which canthen allow the user to use the dashboard screen while driving. The useof hand or fingers or objects held in the hand to operate such a laptopand screen combination has been disclosed in the co-pending references.Clearly one could use normal mice, keyboards and the like which couldplug in the laptop, or directly into a dash receptacle. Alternatively,or in addition, voice commands can also be used.

FIG. 8

Shown in FIG. 8 are user assisted displays used to control machinevision systems associated with TV camera in puts of conditions externalto the vehicle. Two embodiments are shown, a stereo vision based systemcapable of determining range to an object in the z direction, as well asits xy location, and a driver designated feature search, using a screenof the invention.

As shown TV color camera pair 801 and 802 are positioned in closeproximity to the windshield 805 with a baseline B between them toprovide range information as to range R of objects in front of thevehicle when images from each camera are matched in computer 810 usingknown stereo photogrammetric triangulation principles. (see for example,A paper by Dr. H. F. L. Pinckney entitled Theory and Development of anon line 30 Hz video photogrammetry system for real-time 3 dimensionalcontrol presented at the Symposium of Commission V Photogrammetry forIndustry, Stockholm, August 1978, together with many of the referencesreferred to therein gives many of the underlying equations of solutionof photogrammetry particularly with a single camera. Another referencerelating to use of two or more cameras, is Development of Stereo Visionfor Industrial Inspection, Dr. S. F. El-Hakim, Proceedings of theInstrument Society of America (ISA) Symposium, Calgary Alta, Apr. 3 51989. This paper too has several useful references to the photogrammetryart.).

The color (e.g. red) and shape (e.g. round) of an object such as thetraffic light 808 can be used to help discrimination against backgroundin the scene, as can their location (either above or to the side of theroad—which can be inputted to the system automatically given a GPSsignal to locate what jurisdiction the car is in (such jurisdictionhaving known traffic light positions).

In one embodiment, FIG. 8 a, the computer 810 is programmed to simplylook for red or yellow traffic light signal images, such as 815 and 816in the lower portion of respective image fields 820 and 821 (due to lensinversion), and provide to the operator of the vehicle a signal as totheir presence and distance. Optionally, such signals can be used toactuate the brakes or other functions as well. If only presence isneeded, then a single camera data can give this information.

In an alternate and similar embodiment, the signal looked for is brakelights of vehicles ahead, or tail lights. It is noted that a singlecamera can be used to roughly estimate range to taillights since mostare on relatively similar base line widths (almost car width, and arethis baseline is nearly horizontal at all times). Similarly oncomingheadlights can be detected and range determined if desired in thismanner.

In another embodiment of the invention, the user himself, can designatethings of interest for the camera system to sense and help keep himaware of. For example as shown in FIG. 8 b, consider large touch screendisplay on the dashboard 840 on which a TV camera image of the zoneahead of the car is displayed using one of the screen options available.The driver, with his finger 845 can touch the screen to designate aregion of interest, and the computer camera analysis system can considercriteria in this region. For example, the user could point out a carahead, an intersection with traffic lights, a shoulder of the road suchas 855, a line marker in the middle of the road, railway crossings suchas 852, and so forth. Known machine vision programs such as Vision Bloksby Integral vision corporation can be used to process the TV image datato continue to monitor the position of a crossing gate 850 in crossingand to warn if it begins to go down for example. In this manner Machinevision can be of assist to the driver, without the necessity ofprocessing complex scenes. It is particularly true if the driver can sayintersection, or light, and point to the approximate region of the imageto monitor.

In an auxiliary implementation, the car can further include one or moreinfrared light projectors to illuminate retro-reflectors (also generallyin known locations) of vehicles and highway signs ahead or to the sidesof the vehicle. In this case ir is used to keep from disturbing otherdrivers, and near IR (e.g. 0.9 microns wavelength) can still be detectedby conventional cameras.

It is noted that the driver could alternatively ask the system viamicrophone 860 using voice recognition to look for a certain condition,such as “stoplight”. But voice recognition is not as effective todetermine where to look in the image field.

In this example it is often desirable to find the traffic light which islit, so to speak (e.g. red green or yellow), since the lit one is themost distinguishable. A problem exists that if nothing is seen, it stilldoesn't mean nothing is lit. (not fail safe). Thus operator assist bydesignation is useful, to define a region that surely has something lit,such that the sensitivity can be ramped up until it is detected)

FIG. 9

FIG. 9 illustrates an embodiment building on that of FIG. 4, in whichthe steering wheel 901 includes in its center portion a display andtouch device, such as touch screen 910, also with airbag 915 behind it.As in FIG. 4, the image projector 920 and sensing means 925 for thetouch screen is preferably off the axis of the airbag deployment. Bothare included, with the airbag 915 in housing 930.

In this manifestation, the housing for the sensing means and projectorbecomes larger in diameter than currently in today's steering columns,however Since the touch screen in the wheel can allow virtually allessential controls and functions to be viewed or actuated, thiseliminates the need for many of the switches and levers on the columnneeded today for windshield wipers, cruise control, transmission and thelike, which also pose a safety hazard, as do any switches which can beimpacted by passengers. In this example, there are no protruding devicesof any kind required.

It is not necessary to use a rear projection based touch screen in thesteering wheel, but given current laws, an airbag is virtually requiredin the steering wheel. And too a projection device can be fixed inposition in some cases, while the wheel moves. While this avoidswrapping of wires, it does mean the display would not look right on arotated wheel, without computer compensation of the projection toaccount for same.

Alternatively, or in addition, A camera overhead or to the side such asshown in FIG. 23 as well as referenced co-pending applications also canbe used to input commands from human positions, or small switches knobsor other tactile selection and adjustment means and controls actuated bythe human which are on the steering wheel. For example, in this case ifthe controls are done this way, the touch screen 910, need only be adisplay screen without the touch capability, since control actuation bythe user can be done by use of the cameras as shown, or by conventionaldevices.

FIG. 10

FIG. 10 illustrates an embodiment of the invention where a Screendisplay moves around a touch point using a vector of touch, rather thanwhere the touched point is moved by the person touching to differentpoints on a fixed display. This is radically different than any knowntouch screen operation I am aware of, and is achieved herein by having adisplay responsive to the vector of touch, which allows an urging of thetouched point in a chosen direction to command the display on thescreen.

It should be noted that in FIG. 14, the finger 1410 by pressing in onthe pliable screen outer surface, undergoes resistance which opposesmovement from side to side, but does not inhibit it. This resistance canbe helpful in tactilely determining a direction of movement. For exampleas ones finger touches the screen at a point “P”, there is felt in thefinger is a resistance when one wishes to push to one side or another.As soon as one begins to push, say to the right, the display begins tomove accordingly. But the finger itself need not be moved across thescreen, as the force of the finger against the resistance acts as gaspedal so to speak to cause the display to move.

For example consider FIG. 10 a where a screen 1000 is touched at point“P” by finger 1010. The display is controlled to put a slider right onthe finger point indicative of its position along a sliding controlpath. At a future time, after pushing with ones finger a given amount,the display looks like that of FIG. 10 b. Note that the finger is in thesame xy location on the screen, but the display has moved, and is nowindicative of an increase in the position along the path, and acorresponding increase in a variable, in this case heater blower speed.The previous position of the heater control slider is shown in dottedlines.

This same approach can be followed not only by conventional knobs andthe like, but also by more complex items, such as displayed seat iconsshaped like seats (such as typically found in Oldsmobile aurora andMercedes vehicles). These icons can be pushed in any number of differentdirections, not just x and y related.

Position and orientation of the image can thus be in proportion to theabsolute value of a control variable known—which modifies what the imagelooks like as it jumps to a logical position at the point of touch.

By having a display and associated control function which moves aroundthe point of an arbitrary touch, this embodiment requires much lessvisual concentration than conventional touch screens requiring the touchto track a predetermined fixed display.

It is noted that the operation shown in FIGS. 10 a and b can also beachieved using more conventional touch screen technology, but simplycausing the movement of the display shown between condition a and b tooccur by leaving ones finger on the display for a length of time, saidtime related to the rate of display movement and the rate of change ofthe control settings.

However in this case, the direction of change is not clear, just bytouching the screen at a point. Thus some other indication has to begiven (which the touch vector version provides). This directionindication can come from moving slightly in the direction desired, andthen leaving ones finger sit. Or it can be provided by voice input. Orsome other method.

FIG. 11

FIG. 11 is an alternate embodiment of the invention using the image of ausers body portion or object, using light from a projection source, orfrom inside the car to back illuminate said portion.

For example, consider, finger 1100 of a user touches screen 1102 whichis sufficiently transparent (though still able to scatter lightprojected on its surface). TV Camera 1105, located behind the screen andout of the way of the displayed image projector device 1110, is used toview the screen 1102 from the rear. The Camera image is processed insystem computer 1107.

In a first case, the camera sees the darker image of the finger 1100touching the screen, when back illuminated by light 1111 from the usersside (in this case light inside the car. This works well during daylightand in relatively well defined situations, but poorly at night unlessauxiliary lighting is provided inside the vehicle (e.g. From IR LED'swhich can be placed for example in the roof of the vehicle behind thefront seats, which are not disturbing as they are invisible). The camerais ideally used to see the finger touch location in x and y screencoordinates when the projector source is dark eliminating backgroundnoise from the screen. The projector can in some cases be switched offor its displayed image darkened only momentarily, to avoid theimpression of it being off, which can disturb the user.

As an alternative solution to the night illumination problem, a reversesituation can be used, and is generally preferred. Here, in oneembodiment, the finger tip 1101 is front illuminated through the screeninstead using light 1115 from the projector 1110 (or optionally by aseparate source such as IR LED's 1116 (dotted lines) located behind thescreen, whose light, like that of the projector, passes through thescreen from the rear. The reflection 1120 from the finger, is sensedthrough the screen by the camera 1105 as shown.

Given the screen scattering properties, generally required to make theprojected display image visible to users in variant positions, theuser's finger (or fingers) need be close or touching the screen for bestresults. (when touching it may act to frustrate the scattering effect,and work even better).

For best results in the presence of strong projection images, antireflection coating 1121 is applied to the back of the screen to preventbackscatter reaching the camera from the projection source, and thecamera is purposely located off the angle of direct reflection off thescreen from the source as well. If a separate quasi monochromatic sourcesuch as IR LED 1116 is used, a band pass filter at that wavelength suchas 1126 (dotted lines) can be placed in front of the TV camera 1105 inorder to pass largely light from the special source, and not from theprojector (which could also include a band blocking filter at thatwavelength-desirable in most cases anyway to limit heat reaching thescreen.

It is noted that if the screen is to the right of the driver, as itwould be if located to the right of the steering wheel on the dash orcenter stack, the finger will generally approach the screen at an angleas shown—this can make it easier to discern the point of finger contactas a longer stretch of the finger edges can be used to perform thecalculation from the TV image.

The screen if desired, could have raised or indented portions, asdescribed in FIG. 5, such as indent 1130 where the users finger wouldtouch.

The image of finger tip 1101 can be further distinguished by othermethods. For example the camera 1105 is typically a color camera, and ifthe light projected by projector is white in nature, the color of theimaged light from the finger tip will be flesh colored, and only fleshcolored images can be looked for by image analysis software in computer1107. Since everyone's flesh may be different in color, one can teachthe camera computer system during a setup phase, by simply putting onesfinger on a square or squares on which white (or another color) isprojected. It is thus desired to match the return from one or moreprojected colors with an indication of the persons finger.

Another means of distinguishing ones finger is by image sharpness. Sincethe surface of the screen 1102 is typically scattering, it is clear thatobjects that aren't in direct contact with the screen will be seen lessclearly—and in fact become undiscern—able if the finger or other objectis too far from the screen in the direction away from the camera. Thus acriteria for determining finger presence on the screen is both sharpnessand color, as well as degree of light return and size (most fingertouches being within a size range on the order of 10 millimeters on thescreen).

To avoid having the camera system exposed to images which don'trepresent a touch, it is possible to sense independently that a touchcondition has occurred, for example with piezoelectric transducers suchas 1180, and 1181 whose information is processed by computer 1107 andthen use this signal to cause the camera computer to analyze images onthe screen. When a force or other touch condition is detected the systemis programmed to look, minimizing the chance of false signals due tounusual lighting conditions.

Another variant is to see the deflection of the screen or otherindicator of finger location, and using this knowledge, then localizethe search for the finger image. Conversely, one can get a rough fingerimage, and then localize the location of search thru other means, whichcould be acoustic, optical or whatever.

FIG. 11 b illustrates an annular groove 1155 in screen 1102 which can betraced around with finger tip 1101 in order to guide finger tip to thedesired clock wise position indicative of a variable or quantitydesired. The finger image, is discerned best when the finger is fully inthe groove, as it would be when one was “feeling” for same and guidingones finger around it, in a fashion similar to a old style telephonedial, or knob.

FIG. 11 c illustrates a thresholded finger image 1190 as seen by camera1105 when the finger is in direct contact with a outer scattering screensurface, while finger image 1191 is that seen with a finger 3 mm awayfrom the scattering surface. In each case the images are thresholded tothe same intensity value.

Note too that the edge contrast as illustrated in the intensity profile1195 of image 1190 is much higher than the corresponding profile 1196 of1191, which contrast can also be used, besides size at a given thresholdintensity value, to discriminate the two conditions.

While the discussion above has been concerned with finger or otherimages illuminated with light in the visible or near IR wavelengths,alternatively, the Self generated radiation of the body can be used,detected with IR TV (eg pyroelectric) cameras 1105 operating in longerwavelength regions.

FIG. 12

FIG. 12 is an alternate embodiment of the invention using a SAW typetouch screen 1200, such as a Mass Multimedia/Elo brand, model M14-SAWemployed with an LCD display 1201. (note it can also use themicro-display projection unit of FIG. 2). And the display screen may beequipped with eschelle prisms, rectilinear screens, or other optics1210, to direct the light preferentially, for example toward the driveror in another direction as desired.

FIG. 12 b illustrates another form of ultrasonic based touch screen, inwhich an acoustic imaging transducer array 1230 is located behind an LCDdisplay screen 1235 having glass cover 1236 is capable of detecting thelocation of a finger touch 1240 on the front of the screen, 1236. Thearray 1230 is composed of 100 individual acoustic transducers(individually not shown for clarity,) which are read by signal generatorand receiver electronics module 1245 switched to each position insuccession by multiplexor 1250. Switching time for 5 Mhz signals fromeach transducer is about 0.1 seconds, but can be less.

This embodiment can produce an acoustic image of the surface of thescreen, which as disclosed above can have ridges on it, or other tactilerelations as desired. The finger position and degree of indentation ifdesired of the glass plate can be determined by computer module 1260which processes the time of flight of the ultrasonic wave and looks forchanges in the norm. Even though the signals from the various layers ofthe LCD screen are complex, the change in reflected signal due to thefinger touch on glass 1236 is clearly discernable, and pinpointslocation in xy and distance z. For example in this case shown in FIG. 12c the signal back from transducer A and transducer B is different, withtransducer B being stronger, indicating that the finger touch 1240 iscloser to B than A. These signals can be calibrated to give a lookuptable which allows one to compute the xy location of touch in thecomputer 1260. Typical beam spreads of the acoustic send and receivemodules are 20 degrees.

It should be noted that the xy location is not just limited toresolution of 10.times.10 where 100 transducers are used, sinceinterpolation between signals of adjacent transducers can be used toisolate the position to at least 4.times. better in x and y. This givesa 40.times.40 matrix, sufficient for many touch screen applications incars.

Ultrasound can also be used in air in such a manner to monitor a screensuch as in FIG. 2, illuminated by a projection display for example.

FIG. 13

FIG. 13 illustrates a related embodiment of the invention in which asimple LCD display 1300 is located in the armrest portion 1305 betweenthe two front seats (or optionally between two rear seats). This displaymay optionally be equipped with touch screen capability.

The display is positioned to be easy for the driver 1328 to use with hisright (left in the UK and Japan) hand, and can be seen by him by lookingdown and to the right. The touch screen or other display may optionallybe rotate-able so that the passenger can use it as well. When thedisplay is rotated, the display may be varied to meet applicable safetylaws, for example by not presenting graphics or perhaps internet surfinginformation to the driver while moving.

In a preferred example of an embodiment with many desirable advantages,TV camera 1320 in the roof, for example obtains an image of the touchedscreen and the finger 1310 touching it (in essence creating a touchscreen, by noting the position in the camera computer 1321 of thedisplay as well). This image 1330 is then displayed for the driver onthe windshield 1340 of the vehicle in or near his normal line of sight,using a heads up display (HUD) employing holographic or other lenstechnology known in the art. This provides the driver a very desirabledisplay of what is happening, so to speak on the display/touch screen,along with his vision of the road. Its noted that this approach workswherever the display, touch screen or pad exists—as long as the cameracan obtain its image.

For those touch screens or pads capable of outputting the locations ofthe human finger(s), the display can create these without the camera, byusing the image drive circuitry of the touch screen to drive the headsup display as well.

For example FIG. 13B illustrates what a direct image taken with anexternal camera such as 1320 would look like when projected. The personis touching point a virtual slider 1360 for heat control in thisexample, which is moved by the driver to position B, from an initialposition A. This is something like one sees with ones eyes, when oneworks a touch screen type display normally—i.e., you see your hand andfinger near or at the screen. In either case you can get a much betterfeel for what you have to do to make a reading register successfully, bywatching your finger approach the point in question, and watching thedisplay say change color (if programmed to do so) when you correctlytouch it. Alternatively, in other embodiments herein an acoustic signal,or force pulse to your finger can also or alternatively be used toindicate correct contact so to speak. This issue though, is how do youapproach the point, with only a glance, at the screen, and withouttaking your eyes off the road for anything other than an instant.)

This concept is different than conventional touch screens which do nothave an auxiliary display (in this case via an HUD, or “heads up”display) displaying not only the results of the touch in a box on ascreen, but to actually display the position of ones finger on thescreen with the screen information. This is to allow one to “See whatone is doing” without taking ones eyes off the road—a new concept in itsown right, which could be extrapolated to other tasks in the car—or inother applications such as the home or business as well. Indeed theauxiliary display, could be the only display. The touch screen itselfcan be of the standard type such as in FIG. 12, or preferably with oneor more tactile features as in the invention herein disclosed.

It is noted that because of the unique properties of the heads updisplay of such touch screen inputs, that one can maintain aconventional instrument cluster ahead of the driver such as in FIG. 1 ifdesired while still having all the advantages of the touch screens ofFIG. 1 and more besides. Data entered from the touch screen can be usedto override conventional controls if any which are still present. Orpriority can be given to the conventional controls.

Note that to make it easier for the driver, and use up less space on thewindshield, one might only display portions of his finger and theslider, not the whole screen image acquired by camera 1320.

FIG. 14

FIG. 14 illustrates another embodiment of my distortion touch screeninvention As discussed above A TV or computer display image projectorsuch as a Sharp brand LCD based projector 1401 can be used to project adesired display image on the screen 1403. In this embodiment, distortionof the screen material 1403 (for example a plastic sheet with scatteringproperties on its front surface 1405) occurs primarily by virtue of thecompressible supporting material 1406 that allows the relatively thinsheet member 1403 to locally distort or otherwise be changed underaction of a touch, for example by finger 1410,

As shown, the screen member 1403 is separated from rigid Plexiglasbacking member 1415 by refractive medium 1406 of thickness t, which iscompressed or displaced by the force of finger 1410. Alternatively, themedium 1406 can be a liquid such as oil, glycol, or water, that iscompressed or displaced by the finger push (raising the surfaceelsewhere).

Light 1430 from light source 1431 from the rear passes through backingmember 1415 and is reflected off the back surface 1404 of screen member1405. Desirably, the index of refraction of member 1415 and material1406 are closely matched such that little refraction occurs, Ideallyanti-reflection coatings eliminate most of the reflections from thevarious surfaces other than that from the back surface 1404 of member1403 desired.

Light reflected from surface 1404 reflects back toward camera 1435(after retro reflection by expansive retro reflector 1440 made of scotchlight 7615 glass bead material) for example effectively used todetermine the distortion of 1403 due to the retro reflective based “Dsight” image effect or another optical phenomena such as disclosed incopending reference 1 and 2.

Its noted that the Dsight system operates independently of theprojector, and is not affected by its operation. If desired, lasersources instead of a conventional light source 1431 can be used, withappropriate filters on the TV camera to assure function. This holds trueof all optical methods such as grid projection that might alternativelybe used to determine the change in screen 1403.

It is also noted that in another embodiment, the material of 1403 mayitself be in contact with the rigid (relatively at least) member 1415.In this case the material of 1403 should be compressible, and closelymatched where possible in its index of refraction to that of 1415. Inthis case the reflection comes from the front surface of 1403 facing theuser. The refractive medium 1406 in this case doesn't exist, or can beconsidered of t=zero.

Also illustrated in FIG. 14 is the use of a force feedback signal to theusers finger in which an acoustic wave in the screen signals to theusers finger that an event has occurred It can be pulsed, or of staticor varying repetition rate or frequency, or of varying amplitude and isan alternative to providing the user a beep tone for example to indicatethat an action has registered.

As the user finger 1410 presses on the screen 1403, a strong acousticwave 1465 is generated by transducer 1470 in material 1406. Thetransducer input is triggered by the sensor of what ever type is used indetermining the screen has been touched (in the case here, it issignaled by computer 1446 analyzing the image from TV camera 1435 whichdetermines a touch at a given screen coordinate has occurred (and itsmagnitude if desired. This then is compared to the screen projectiondata, to determine what the touch has registered, if anything. When somesort of registration data is desired, The acoustic wave is generated andimpacts finger 1410 which then feels the event. The degree of touchingin the z or force direction can be used to control the amplitude or thefrequency of the acoustic wave 1465 and thus signal to the user thedegree of z actuation.

It should be noted that the finger 1410 by pressing in on the pliablescreen outer surface 1405, undergoes resistance from sidewalls 1411 and1412 for example of the indent, which opposes movement from side to side(and similarly, up and down), but does not inhibit it. This resistancecan be helpful in tactilely determining a direction of movement. It canalso be actually increased or enhanced in some manner by using anacoustic source to provide a pulse or other signal to the finger inproportion to movement or direction, generated in the medium 1406 byacoustic wave sources such as 1470.

A useful display with such as screen is when the display is caused to“jump” to the point touched, using that point as a new reference for thedisplayed data (eg a heat control bar). This is discussed relative tofigures below

FIG. 15

The TV camera based rear projection touch screens and other user signaldetecting aspects of the invention allow one to use a laser pointerindication on the screen as well for control, entertainment or otherpurposes. FIG. 15 illustrates a method according to the invention usinga laser pointer 1501 which is aimed by user 1502 at the touch screen1510 on the cars dashboard 1518 producing spot 1515 or other indication.This signal on the backside is visible to TV camera 1520 which may bedesirably equipped with a band pass interference filter known in the artcentered on the laser pointer wavelength (e.g. 670 nm), 1521, toincrease signal to noise. This can be easily combined with the samecamera sensing touch distortion, if the light illuminating the screenfor this purpose 1530 is also from a laser or other source of the samewavelength (assuming the filter is used). Use of lasers helps in highsunlight environments in the car, but is not necessary for operation ingeneral.

Computer 1550 processes the image from camera 1520 to determine the x,ylocation of the spot 1515 on the screen and any other data such the spotshape or coded signature, etc, as disclosed in my copending applicationSer. No. 09/568,554 incorporated by reference

FIG. 16

FIG. 16 illustrates further camera based embodiments of the invention,several of which are discussed as well in other forms in the copendingreferenced applications above.

For example, these applications and previously granted patents by theinventor such as U.S. Pat. No. 5,982,352 have discussed the use ofcameras either single, or stereo pairs and the like to view persons todetermine positions and motions of their extremities or objects held bythe person. When data is determined from such activity to be of concernfor example, alarms can be provided (if out of position occupants aredetermined say), or to signal Onstar (GMs name for a navigational andhelp aid).

As pointed out previously, such camera inputs (eg from camera 701) as torotational position of a head such as 710, can cue the appropriatescreen display to moves as head moves—not just for viewpoint changes,but to help the person view the display (for example If one is looking abit to the side, the critical data on the instrument cluster ahead ofthe driver can be shifted slightly sideways.) In addition, as pointedout before, if one moves one head, this could cause the display tochange function, or if rear view images were presented on the display(from rear facing cameras for example), the display 1600 could changeviews of the cameras.

For example, with cameras having large numbers of camera pixels at lowcost, a single camera with for example 2000.times.2000 pixels such as1601 looking rearward from above the rear window 1602 as shown, may beabove to cover with a suitable wide angle lens 1605, the total angularfield of interest “K” behind the vehicle 1610. However what is displayedon a screen on the dash 1625, for example in zone 1630 allotted for therear view camera image, might be only a fraction of the field (ie only afraction of the pixels displayed at once corresponding to an angularregion less than K)—and this swept, by simply turning ones head (just asone does when one looks in the rear view mirror).

To aid camera viewing of the human, it is often desirable to have retroreflective or other discernable marks rather than arbitrary humanfeatures. For example, glove 1640 with targeted fingertip 1645 and thumb1646.

As taught in the copending applications referenced, similar cameras canalso be used to inexpensively see objects which the human moves forsignaling purposes or to cue the control system, for example as to likeswitch positions, or heater vent locations, or stalk positions etc. Forexample one could have interchangeable dash overlays, not only forcertain screen functions or markings, but also sliders, raised or otherwise which could be optically sensed by TV cameras, for example with thecamera 701 of FIG. 7 located in the headliner.

It should be noted that where features on the human can be discerned bythe computer and camera system, that such features themselves canprovide the input. This too can include the fingers and hands, etc. Suchfeatures can also be watched for safety purposes, to assure a door isn'tslammed on a child's hand for example.

Also pointed out in copending references, objects in the car can also beviewed by cameras, or themselves have cameras, and communicate withother things in the car like fax machines, phones and the like.

The system can aid recognition of the drivers face, hands, eyes etc. fortheft security or other purposes. The art is full of such things thesedays (See for example Suzuki U.S. Pat. No. 5,859,921 Apparatus forprocessing an image of a face). Such devices can also observe the stateof the driver, to determine driver attentiveness and ability to drive.Note too however, that such may be further augmented using theinvention, by having a keyboard for example screen presentation, inwhich certain commands have to be typed in order to start the car—thesecould assist in determining the drivers state or authenticity (via codestyped in).

A camera of the system can also recognize shape of face, andparticularly the pointing direction of the eyes, in order to rotate apresented view on the screen, for example a rear view obtained withexterior mounted cameras (see for example U.S. Pat. No. 5,949,331 bySchofield, et al.), or a night vision view out the front using IRcameras or scanning microwave equipment. Indeed such views, when sensedby the car display and actuation system of the invention to be needed orcalled on by the driver input can be presented on the display 1405 ofdash along with the minimum amount of normal data (speed etc) needed,for example as expressed by a digital speed indication.

In one embodiment, discussed above, when the user is seen by a camerasuch as 701 and associated image processing computer to be looking in arearward direction, for example by turning ones head slightly, the dashinstrumentation may be programmed to immediately convert to a mode wherethe rear display is presented as shown on the dash.

FIG. 17

The disclosed camera based object position sensing and control system ofthe co-pending applications incorporated by reference can be used toturn any part of a car into a control mechanism without wires or leversetc. Detected can be common parts of the car in any position where aconveniently mounted camera(s) can get a reliable view—vent positions,turn signal indicators, shift levers, knobs, buttons, ashtrays footpedals, etc. Isn't just the point you actuate. Its also a point you moveto a new position. For example If you move a vent to a new angularposition. Knobs, window and lock switches, etc. seat positions, all canbe camera sensed. Thus such controls can be inexpensive, as there isnothing that need connect to them, as all transduction is noncontact—and there is an economy of scale of multiple things in field ofview of the camera(s).

For example. Consider overlay 1700 on flat panel display screen 1708.The overlay contains knob 1710 with pointer 1715 As shown in FIGS. 1 and2, such an overlay on the screen can be read from the rear or projectionside (if a projection display is used). But alternatively an overlay canbe constructed to have a pointer or other marker on the front of theknob as shown, for observation in a front viewing mode.

For operation in a front viewing mode as shown, the persons hand (notshown for clairity) moving the knob, slider or other control can get inthe way of the camera, 1720 used to see the knob and its pointerlocation at least for some camera and person positions. For this reasonit is desirable that a certainly latency be acceptable. For example, ifone moves the heater slider to a new position hotter than the last, itis generally alright to have the sensing of the new slider position beperformed after the hand is retracted (and thus not obscuring the view).This can be sensed by the camera 1720 and its computer system 1740, as asituation where a standard view or “image signature” of the slider 1750(or other object to be sensed) is present and not blocked.

For example If in the field of view of the camera, the slider mark 1765occupied an area of 12 dark pixels in the processed camera image 1760surrounded by bright ones (which could be the case if it was a dark markon an overlay on a backlit touch screen), and such a situation wasn'tseen, then no action would be taken until the situation once againappeared. At that time the new position of the centroid say, of the 12pixels would be compared either to the absolute position of the dash, orto the previous position of the slider, and appropriate action taken.(this same logic also works for rear viewing as in other embodimentsabove, where the problem is not necessarily obscuration, but noise fromprojected image variations in an instant image situation).

The processed image of the knob pointer 1770 is shown as well.

Most controls in a car today fit this scenario. Only some things likeadjusting something in real time to “get it right” with no other cueswould require constant monitoring of control position.

FIG. 18

FIG. 18 illustrates a ball joint or gimbaled physical selection oradjustment means in which position and orientation of the handle inputis determined optically using for example TV Camera 1800 to view a fourpoint target set 1805 (two targets shown for clarity) on ball member1810 secured in ball joint clamp 1825 Joystick 1820 is attached tomember 1810.

Using the invention of Pinkney et al, U.S. Pat. No. 4,219,847, or thedevices of the El-Hakim reference, 6 axes of position and orientation ofthe member 1810 can be determined using program in computer 1830connected to the TV camera. This axes data can then be used by computer1830 to determine a control function desired (e.g. mirror rotation) andan appropriate display or voice or other response if desired. The cameraused in this case, can if desired be the same one used to see knobpositions, if such is used.

Of considerable interest however is that this system can be mountedright to the dash, even to the display screen 1850 itself, as shown. Inthis case data selected by any of the up to 6 degrees of motion inputtedby the user, can be displayed right next to the joystick by control ofthe display means such as a computer controlled projector projectingdata on the display screen.

If the ball joint is sufficiently tight, the joystick will stay where itis positioned. Alternatively it can be spring loaded to return to homecenter position, for example. Or it may be affixed with force feedbackcapability known in the art (eg from Immersion corporation) to provide ahaptic response to the user as well, indicative of actions taken, ordata to be fed back as a result thereof.

Note that such a physical selection joystick device can also be built inthe form of a palm resting ball or other. And such devices can be usedas game controllers—in the car or home, for example

FIG. 19

The invention can be used in the home or work, besides in the car, aswas previously disclosed as well. For example either the on axis or offaxis version (FIG. 2) of the projection device can be used, the off axisbeing more complex, but able to occupy a bit different space (realizingthe on axis version can be used with mirrors to make it taller ratherthan deeper, just as projection home TVs are today. Since the displaysurface, if build of the surface distortion type, can be chosen fromdifferent materials, one could use it as a home black board or corkboard, putting family notes on, keeping records, drawing pictures ordiagrams, etc.

The advent of high definition projection sources, and large computermemories at low cost, allows one to project images of fine detail from avariety of sources. This HDTV type of image capability, in turn makes itpractical to get close to a large screen projection, which here to forehas been too crude (pixilated) to view at close range. At this point theprojection TV becomes essentially possessing the detail of a finephotograph, or drawing.

Clearly this is useful on the car dash. But interesting too is in thehome where such resolution makes it possible to display pictures scenesand the like, still or moving, in the sense of interior decoration.

Shown here is an All Purpose touch screen display for the home,incorporating controls for heating and lighting and appliance systemsplus entertainment and internet features, plus interior decorating andother features.

As shown a large screen HDTV (High definition television) capabledisplay 1900 is on wall 1901. It is constructed in this case using aprojection micro-display, in a manner similar to the figures above. Forthe moment, it is shown with the display on axis, but it could be offaxis like FIG. 3 as well, which could help in some cases to make thedisplay less deep. As can be appreciated from the above disclosure, thisdisplay could at one time display beautiful pictures, say of a scenefrom Venice in the 1700's (static, or moving) to decorate a Venetiandecor living room for example, while at the next moment provide a touchscreen to act as home control center, for various functions such asheating, air conditioning, lighting, burglar alarms, appliances, and anyand all other desired home functions.

Then at the next moment, it could be used for entertainment (e.g. DVDdiscs, TV entertainment) or to communicate with the internet. Evenimages for the interior decoration can be downloaded from the internet.For example If one wanted to have a Venetian masquerade party, one coulddownload from of the internet Carnevale pictures of a real party of thattype in a Venetian setting, which could be sequentially played duringthe guests visit.

In addition, one could use it to write on, for example notes to otherfamily members could be entered directly just as one might do today witha blackboard in the kitchen. The written images, could be transmitted aswell to other locations in the home, or to family members in the field.

The screen can also be used for games, as has been discussed at lengthin other copending applications. Camera 1920, or stereo Camera pair 1921and 1925 located near the display can be used to sense inputs fromhumans or objects in the room.

If the display were large sized, covering a good portion of a wall, whatwould be the effect? The use of such displays was discussed at length inthe referenced Co-pending applications, for CAD Systems, Games,Education, and the like. Of particular interest is Life sized displaysto give a “natural” feel to the data displayed. People, if life sizedisplayed, become “Guests” in ones house, so to speak. One can in anatural manner use ones hands to wave at them (sensed by cameras of theinvention too). And one can use the computer and screen to provide tothe people in the home life sized digital images of themselves invarious clothes they might wish to buy over the internet for example.And they can modify those images using a computer, to suit how they feelthey might look under different weight reduction or enhancing regimensfor example. Such digital data as to their initial shape can be obtainedusing laser triangulation techniques disclosed by the inventor and hiscolleagues as mentioned in U.S. Pat. Nos. 5,362,970 5,880,459,5,877,491, 5,734,172, 5,670,787 and others.

It isn't just the home where such a combination control and imageentertainment function could be of use. Ones office is another examplewhere the room functions, and pleasing decor, and perhaps ones owncomputer functions can all be combined. A desk of this type in thecontext of a CAD terminal is shown in co-pending application U.S. Ser.No. 09/435,854. In this case too, one can write on the screen forpresentation purposes, or to transmit such data to others via a networkfor example. One can display pictures or drawings, and alter them andtransmit or store the altered data as well. Such a display screen in ahome or office, could become the main focal point for activity, since itallows all functions of a computer to be used, (and could have a plug inkeyboard or mouse if desired), as well as entertainment and the like.

The life size aspect of such a potential screen in a business contextmay relate most to dealing with objects that ones business is comprisedof. For example in the context of car parts, the thing has a realdimension.

But the real benefit is dealing with people, who can be portrayed actualsize. Or in dealing with things people use or interact with, which canalso be of real size relative to the people in question.

A good example is the “Digital You” “mirror” of the fashion relatedco-pending application, in which the touch screen—or even just plaindisplay screen, is in the form of a full length fashion mirror forclothes trying. you can also have a perfect ballerina model, executing aperfect digital move of a particular ballet step.

It is also possible to present to a user, a life size surrogate personfor interaction, in this case in the context of clothes design where onetries different designs on a digital person, and touches them just asdescribed in the copending application—in this case however, not toalter the garment, but to actually change its original designdimensions. And not necessarily fit related, put perhaps style related.

FIG. 20

FIG. 20 illustrates another advantage of the invention, namely theability to locate the touch screen in areas of high electrical problemor contamination. In this case, it is in a shower bath 2000, wherelocated at end 2001 of the shower opposite the shower head 2005 is atouch screen 2010 of the invention, located in the wall of the shower.In this case the unit is used to allow the user to access the internetand all desired audio visual sources while in the shower, but at thesame time by selection of a different screen it can control thefunctions in the home as described in FIG. 17, to include the plumbing,and the various features (heat, pressure, pulsation, etc) of the showeritself.

The touch screen can operate at low voltage but of interest is that thetotal electrical part of the screen can be shielded easily from theusing part attached to the shower. For example consider rubber gasketmaterial 2040 which isolates the electrical components such asprojection section 2045, computer control section 2050 from screen inputsection 2053—in this example a direct camera view 2058 of a screen face2054 having sliding input or in this case rotating knob 2055, androtating knob 2057 viewed from the rear as disclosed in FIGS. 2/3above).

For point of illustration, a completely different screen face 2065having other knobs and input devices such as pushbutton 2068 and 2070respectively, can be interchanged with the screen face 2054 as desiredby the user. In one preferred embodiment, the user may reprogram thesystem to have these physical functions represent what ever variables hewishes.

FIG. 21

FIG. 21 illustrates an additional embodiment of the invention directedat optically sensed actuation of screens, even by pointing at them.Direct detection by one or more TV cameras of the pointing vector ofones finger (or other object) has been discussed in co-pendingapplications. A laser pointer application has been described above andin co-pending applications as well. However in this case, ones finger isviewed through the screen of a projection display, which for the momentin a car application is assumed to be in a location not requiring anairbag application though the screen (which however could be affected byangling the camera and sensing system here disclosed as in FIG. 3, so asto be out of the way of the airbag deployment.

In this example, let us assume that a stereo pair of cameras 2101 and2102 is located as shown in FIG. 21 a, which conveniently is outside thedeployment path of the airbag, if present. If the screen 2105 wasperfectly transparent, this would be the same arrangement shown in thenumerous examples for computer inputs using human features and objectsshown in my co-pending applications, where the cameras are lookingdirectly at the user. However, what if the screen is diffuse, as itwould have to be to some degree for use in conjunction with imageprojector 2120, in order that viewers could see the informationdisplayed on the screen?

For objects such as finger 2130 located a distance “d” from the screen2105 as shown, I have found that it is possible to obtain useful x,y,and in some cases, z (depth direction) informational data concerning thefinger or other object such as a pen tip, even if the screen is diffuse,which is a novel result of the invention (discussed also in FIG. 11). Itis noted that an IR LED or LED Array, operating at for example awavelength of 0.9 um, 2140 can be used to illuminate the users finger orother object through the screen. If desired, band pass filters 2131 and2132 passing the 0.9 um radiation can filter light reaching the cameras2101 and 2102 so as to substantially block light from the projectorsource (which also could include a beam blocking filter 2121 at the samewavelength, or even more preferably, to block all IR outside the visibleregion from the projector lamp (thus reducing heat on the screen andremaining portions of the device).

Alternatively, in another useful arrangement, the screen 2105illuminated by projector 2120, is preferentially scattering at eachelemental point on the screen in the lateral, or “x”, direction acrossthe vehicle (or across a room in the FIG. 20 case), but less diffuse ifat all in the vertical direction (out of the plane of the drawing). Thisis the case if a rectilinear screen with vertical diffusing lines isused for example.

If in this case, the stereo pair of cameras 2101 and 2102 are spacedwith their baseline in the vertical direction out of the plane of thedrawing, then they have an substantially unmodified view of the objectsituation in that direction, and accurate Z locations can be determined(and y data as well—though x data is modified by the screen effects).

Another variation utilizes a screen 2105 equipped with a DigilensCompany or other type switch able device 2155 (dotted lines) to controlthe holographic diffuser 2160 (dotted lines). On command of computer2165, the diffuser switch and the projector 2120 are switched off, andthe cameras 2101 and 2102 look directly through a now transparent screenat the user (typically hands or fingers or head), 2130. If the switchingoccurs every 1/60 sec or so, flicker of the image to the eye is minimal.The cameras output is analyzed by image processing program computer 2165to obtain a stereo image of the edges of the object such as users fingerhead, or the like and the output made accordingly to control thefunctions of the vehicle and of the display as desired

FIG. 22

The above control aspects can in some forms be used in a car as welleven with a small display, or in some cases without the display.

For example, consider car steering wheel rim 2200 in FIG. 22. Inparticular, consider hinged targeted switch 2210 (likely in a cluster ofseveral switches) on or near the top of the wheel, when the car ispointed straight ahead, and actuated by the thumb of the driver 2211. Acamera 2220 located in the headliner 2222, and read out by microcomputer2225 senses representative target 2230 on switch 2210, when the switchis moved to a up position exposing the target to the camera. (or onecould cover the target with ones fingers, and when you take a fingeroff, it is exposed, or conversely one can cover the target to actuatethe action).

The camera senses that target 2210 is desired to be signaled andaccordingly computer 2225 assures this function, such as turning on theradio. As long as the switch stays in the position, the radio is on.However other forms of control can be used where the switch and targetsnap back to an original position, and the next actuation, turns theradio off. And too, the time the switch is actuated can indicate afunction, such as increasing the volume of the radio until one lets offthe switch, and the target is sensed to have swung back to its originalposition and the increase in volume thus terminated.

In operating the invention in this manner, one can see position,velocity, orientation, excursion, or any other attribute of actuationdesired. Because of the very low cost involved in incremental additionsof functions, all kinds of things not normally sensed can be. Forexample the position of a datum on a plastic air outlet 2241 in thedashboard of FIG. 1 c can be sensed, indicative of the direction ofairflow. The computer can combine this with other data concerning driveror passenger wishes, other outlets, air temperature and the like, toperfect control of the ambiance of the car interior.

It is also noted that the same TV camera used to sense switch positions,wheel position, duct position, seat position head rest position and avariety of other aspects of physical positions or motions of both thecar controls and the driver or passengers. And it can do this withoutwires or other complicating devices such as rotary encoders whichotherwise add to the service complexity and cost.

When the camera is located as shown, it can also see other things ofinterest on the dashboard—and indeed the human driver himself, forexample his head. This latter aspect has significance in that it can beused to determine numerous aspects 1. The identity of the driver. Forexample, if a certain band of height isn't reached, such as point P onthe drivers head, the ignition can be interlocked. Much simpler thanface recognition, but effective if properly interlocked to preventrepeated retries in a short time period. 2. The position of the head ofthe driver in case of an accident. As detailed in reference 4, a cameraor cameras can be used to determine head location, and indeed locationof the upper torso if the field of view is large enough. Thisinformation can be used to control airbag deployment, or head restposition prior to or during an accident (noting too that headrestposition can also be monitored without adding any hardware).Particularly of interest is that the pixel addressing camera of theinvention can have the frequency response to be useful in a crash,sensing the movement of the person (particularly severe if unrestrained)within a millisecond or two, and providing a measure of the position forairbag deployment. Additional cameras may also be used to aid thedetermination.

Using a pixel addressing camera for camera 2220 confers additionaladvantages. For example consider the image of the car interior In thefirst instance one can confine the window of view of a certain group ofpixels of camera to be only in the region of the steering wheel. Thisallows much faster readout of the more limited number of pixels, andthus of the steering wheel switches, at the expense of not seeinganywhere else in that particular reading. But this may be desirable insome cases, since it may only be required to scan for heater controls orseat positions, every 10 seconds say, while scanning for other moreimmediate items a hundred times per second or more. A good example aresafety related functions. 5 per second might suffice for seeing wherethe turn signal or windshield washer control was, as an example.

Scans in certain areas of the image can also depend on informationobtained. For example one may initiate a scan of a control position,based on the increasing—or decreasing—frequency of an event occurrence.For example If the persons head is in a different location for asignificant number of scans made at 15 second intervals for example,then in case of a crash, this data could be considered unreliable. Thusthe camera window corresponding to pixels in the zone of the head couldbe scanned more frequently henceforward, either until the car stopped,or until such action settled down for example. Such action is often thecase of a person listening to rock music, for example.

Similarly, if someone is detected operating the heater controls, a scanof predominately heater function controls and related zones like airoutlets can be initiated. Thus while normal polling of heater controlsmight be every 2 seconds say, once action is detected, polling canincrease in the window(s) in question to 40 times per second forexample.

Scans in certain areas of the image can also depend on informationobtained in other areas of scan, or be initiated by other controlactions or by voice. For example, if hard de-acceleration was detectedby an accelerometer, but before a crash occurred, the camera couldimmediately be commanded to begin scanning as fast as possible in theregion of the image occupied by the driver and/or any other humans inits field of view. This would be for the purpose of monitoring movementsin a crash, if a crash came, in order to deploy an airbag for example.

While illustrated on the steering wheel where it is readily at hand, itcan be appreciated that the position of switch or indication devices forthe purpose at hand could be elsewhere than the wheel, for example on astalk or on a piece of the dash, or other interior component—indeedwherever a camera of the invention can view them without excessiveobscuration by persons or things in the car. It need not be on a careither, controls of this type can be in the home or elsewhere. Indeed aviewable control datum can even be on a portable component such as oneskey chain, phone, or article of clothing apparel, or whatever. Similarlythe camera 2220 can view these items for other purposes as well.

The teach ability of the invention is achieved by showing the camera thecode marker in question (e.g. a short bar located on the wheel), and inthe computer recording this data along with what it is supposed tosignify as a control function—for example, turn rear wiper on to firstsetting. This added functionality of being easily changed aftermanufacture is an important advantage in some cases, as for example,today after-market addition of wired in accessories is difficult.

FIG. 22

FIG. 22 illustrates a keyboard embodiment of the invention, in this casein the car touch screen, but certainly not limited to car applications.

Taking the embodiments such as FIG. 5 above one step further, supposeall keys were just circular depression type indents in a transparentpiece—either the screen such as depicted here as screen surface 2200(itself for example similar in function to screen 1403 of theinvention), or an overlay thereon, such as member 2201 dotted lines. Onecould have all of the keys of the QWERTY Keyboard for example,represented as indents such as indent 2205 having projected on it theletter “Q” (Alternatively the letter could be just projected on thescreen using the display alone, with no tactile feel). Despite thenumber of keys, the effect on other images presented on the screen (suchas a graphical map of a navigational strategy to reach a location in thecar, for example) would not be unduly affected, as the screen typicallyhas a surface finish which scatters light from its outer surface (plusdirects it in the case of the prismatic or rectilinear screen.

Clearly the goal is to provide, on the dash, or other interior portionof the vehicle such as an arm rest, a keyboard not unlike those familiartoday. This can be used for all the computer input functions known inthe art. But it can also be used for different things as well. Forexample with one or more fingers at once or in succession, one can makea heater do something (Multi-finger at once capability is the keyadvantage of the distortion based screen, not shared with many othertouch screen/pad technologies). One other example is that if a QUERTYkeyboard with 40 indents, say, was used, a separate illumination of theindents might show up heater and air conditioning choices.

For example, consider again screen 2200 which has numerous indents, only6 of which are shown for clarity, 2205 2210. One moment the keyboardletters of the QWERTY keyboard can be projected as discussed above, butthe next moment on command as needed, shown here in the representativecase, the projected numbers are 1,2,3,4,5, corresponding to 5 heaterblower speeds.

The command to change projections, and thus the screen characters andtheir meaning for touch action can be made for example, either by touchof some change area of the screen, or by voice, using voice recognitionto bring up the new screen. Alternatively as discussed elsewhere in thisapplication, for a keyboard is in a car, a camera located in the dash orheadliner, for example, can image and using suitable image processingand interpretation algorithms, determine the position or velocity of apersons hand, finger or head for example to get such a switch command,or at a more complex gesture (movement sequence of the person).

Note that the screen surface, or an overlay thereon, can be changed tocreate different indented or raised styles to suit different user needs.These could be specified when ordering from the factory, or dealerinstalled, or even user interchangeable. They can also incorporatecustomized printed on writing or designs, or contains specially addedfeatures, such as more knobs, or bigger knobs or keys (for elderlydrivers, say). Similarly, the projected images could be bigger too.

It should be noted that the device shown in FIG. 14 provides usefultactile feel in such a keyboard touch screen system. The transparentmedium displaced, creates a force resisting the action of the finger.This force can be increased by use of a pressurizing device, whichdevice can be on all the time, or which may be actuated when the actionof the finger on the screen is sensed by the camera or other sensingdevice used to sense touch on the screen.

In a similar vein, it is noted that rather than create a local slope ona flat screen, one can use the invention in a method by which slope isreduced or removed to provide the indication desired. (in which case,the surface of the screen is flattened out so to speak). For exampleconsider FIG. 22 b which shows a touch screen having a transparentmedium 2240 pressing (in this case upward) against a transparent overlay2245—urged by the pressure of liquid 2250 pressurized betweentransparent rear member 2255. In this case the finger locations areround bumps such as 2260, which are touched and pressed down against theaction of the pressure by for example finger tip 2265.

Touch screens with high resolution and other useful functions are nowdescribed.

FIG. 24

FIG. 24 is an touch screen improvement whereby touch designationprecision of small displayed features is assisted by use of two fingersand/or demagnified cursor movement controlled by ones finger movement.

The pros and cons of various types of conventional computer inputs(mice, trackballs, touch screens etc) are well known. Touch screens aretouted as superior and more naturally intuitive, as you can point rightat the thing desired, such as an icon, word or whatever.

However, in prior art touch screens, spatial resolution (in xy on thescreen face) is often limited due to touch screen resolution, but moreimportantly, finger size and other finger characteristics, such asforce, capacitance, impedance or the like. Even in my distortion basedtouch screen inventions, resolution is difficult to attain if the screendoesn't significantly distort or undergo stress upon a given force oftouch.

So while touch screens are great for input, unless one uses bigcharacters (which are natural human wise in size only on aproportionately big screen), there is a lack of spatial resolution dueto the size of your finger, which limits the size of displayed objectwhich can be touched accurately. Thus in touch screens you see inKiosks, for example, you touch a box, or a big icon—but you seldom areasked to touch to select a letter in a word, as you might do with amouse and screen pointer.

To improve spatial resolution, now disclosed is a significantimprovement on my touch screen invention, and to at least some forms ofconventional touch screens. Importantly, it solves the problem ofdesignation of small displayed data items such as letter or charactersusing “large” fingers operating a touch screen. And it thus allows atouch screen to become an excellent generalized computer input.

There are several manifestations to this. The first concerns use of atouch local to the point desired to “steer” a displayed screen cursor tothe point, in a de-magnified manner in which a large finger movementcauses a smaller cursor movement—often much smaller, e.g. 1:5. Saidanother way, I can first put my finger on the screen local to the lettersay at a point at which a cursor is then displayed near my finger. Ithen can move my finger around at the point of touch to aim the cursorat the letter.

For example, as shown in FIG. 24 a, a persons finger 2401 touches touchscreen 2405 at point 2410 near in both x and y, and perhaps an inch awayfrom a letter “a” that it is desired to designate (typically the letteris in a word which is part of a sentence, none of which is shown forclarity). The letter is typically smaller than the persons finger. Atthis point one of several alternatives may be executed.

In a first alternative, a cursor 2415 (in this case an X shaped cursor)is caused to appear under control of the display control computer on thescreen 2405 a distance of lets say ½ inch above the finger point oftouch 2410. (Using known programming techniques which vary the verticaldeflection of the screen raster, this amount can be user set, if otheramounts are more preferable). A demagnification of movement can beemployed, if desired, in sensing and cursor display system such that thecursor moves slowly either laterally or rotationally toward the letter“a” 2420 desired to be indicated for a given touch movement. In thiscase you would touch somewhat farther away, and “inch” toward the pointdesired. As the persons finger moves to point P′, the cursor has inchedupward and over to the letter “a”. to illustrate, a movement of thefinger on the screen of 0.5 inch in x to the right, would for examplemove the cursor only 0.1 inch (a 5:1 demagnification of movement, makingit easier to move the cursor in small amounts needed to more preciselydesignate small items or precise points displayed.

Shown in FIG. 24 b is a second example, usable on those touch screenscapable of input with two fingers. In this case thumb 2441 touches thescreen, causing the cursor to be established at point P as before. Butin this case finger 2450 touches the screen as well at point “Q”, and bymoving the finger around the thumb point of touch, the cursor is causedto rotate around point p. And by moving finger 2450 radially toward oraway the thumb, the cursor 2455 is caused to extend toward the letter aas desired (the first point of cursor location having been determinedfor example, by the initial spacing “S” between the thumb and forefingeron the screen).

It is also possible as shown that the finger 2450 and thumb 2441 canalso be used to bracket the desired point “a” to be designated, withthen a movement of the finger relative to the thumb can cause the cursor2455 to move along the line 2470 between thumb and finger. When thecursor hits the desired point, one stops the finger motion. For a screenwhich can accept z inputs, one then just can push in with the thumb orfinger in order to register the selection. Other wise one could use avoice command or hit a key or whatever.

Finally, the radically different touch vector approach of FIG. 29 canalso be used for this purpose, but no other touch screens than my ownare known to be able to operate in this fashion.

FIG. 25

To recap, shown in FIG. 24 above are at least the following ways tointeract with a touch screen. 1. Touch at one point and move—see changein local position from starting point, or possibly distance fromabsolute start 2. Touch with two fingers. Move one relative toother—linear or rotationally

FIG. 25 now illustrates further embodiments of the “touch mouse” of FIG.24 using a pinching or other action-in this case with respect toinformation in a spreadsheet cell or a word document word or letter.

As shown displayed on a screen 2505, is a cell 2510 of the many cells ofan “Excel” or other spreadsheet. A user, with a quick move can go rightto the cell of interest and with his thumb 2515 and forefinger 2516,“pinch” the data in the cell 2510, in this case the dollar amount$38.10. The pinching action, for example can be programmed to cause theExcel program, to “cut” the data in the cell, for “pasting” elsewhere.The user can then move it, by sliding it on the screen while pinchingit, to empty cell 2530. When movement stops and the fingers un-pinch theprogram in this case would paste the data there. This move is totallynatural, just as one would pinch any small object and move it tosomewhere else and deposit it. This is considerably faster than using amouse or touch pad, which needs to incrementally move a cursor acrossthe sheet, highlight the item, and then drag it and drop it in the cell

FIG. 26

FIG. 26 further illustrates an embodiment of the invention using forexample a sliding lever, rotary knob or joystick representation on thedisplay screen. This is another variant, where what ever point you firsttouch becomes the origin for subsequent movement of the displayed bar,knob or the like.

When used in a car, a driver can just glance over at the screen, touchthe screen at a first point desired, which then signals the computer toprovide a display on the screen, of a rotary knob, lever or whatever.The display is preferably oriented so that the starting point or frameof reference of its movement is established at the point of fingertouch. Alternatively, one can touch the screen at a first point, andthen indicate by pushing in on the display or other means that the lastpoint the finger reached is to be considered the first point in thiscontext.

Such a display is preferably big so that one can then easily see theinitial position of the knob or slider In the case of a knob, you thencan turn it with ones fingers using a touch screen which has a twistmotion response. (see FIG. 3). As it is twisted in this virtual manner,a movement of the knob dial with twist may be displayed, along withpossibly other changes on the screen such as colors to indicate heat andthe like as desired.

It should be noted that in some cases it may be desirable to have anindent in the screen for one finger or other to locate in. This can behandy when driving if it is desirable that the display of data and otherinformation be in the same place on the screen. Alternatively a smallridge in the screen, including linear or annular one can be used.(discussed also in FIGS. 1, 5 and elsewhere).

For a slider, you just touch and move the lever, with your finger—orbetter pinch it and pull it back and forth as desired in a virtualmotion.

A screen there fore could then provide the ability to at various timesdisplay and actuate touch like dials, knobs, etc on the screen, andstill have them be conveniently the size of a human hand and fingers.Could just display data in a big area—driver just tries to hit anywherein the area and receives an indication back like clicks from acousticpulse or other indicator to his finger.

Such a device can also be used at the side of a page of typewritten datain order to designate a letter in a paragraph say. For example, considerFIG. 26, in which a persons finger 2650 touches point P on the side of aline of writing 2651 on displayed page 2655, and a slider bar 2660 isdisplayed on the side of the page with a lever 2665 also displayed,which acts to slide a cursor 2670 toward a letter “a” to be designated.Often desired for accurate designation of the letter or other object, isthat the movement of the cursor be de-magnified from the movement of thelever.

FIG. 27

FIG. 27 illustrates improved screen distortion and stress based opticaltouch screen embodiments, useful in my previous and co-pendingapplications referenced.

Illustrated, is a clear plastic touch screen 2700 which is reinforcedwith fine steel wires such as 2705 and 2706 supported by rectangularring member 2710 (typically of steel or aluminum) which provideelasticity to the screen and basically carry the load caused by a touchto the screen by finger 2720. This has the effect too of making thescreen not have too much deflection in the middle, often a problem whenlarge forces are used with an unsupported screen clamped only at theedges. Additionally cross wires can be used to further strengthen thescreen such as 2715 and 2716.

An Image projector as before, projects information onto the screen fromthe rear which is however, blocked in small regions by the wires. If theblockage is small relative to the data presented, it is notobjectionable. In addition, or alternatively, the projector control 2730may be imputed with data concerning the wire location on the screen, andthe display program created or modified to keep critical data away fromthese locations, or at least to make data presented at the wirelocations of large enough size as to not be impacted. This isparticularly of use at the wire junctures, such as 2735 where crossedwires intersect.

FIG. 28

FIG. 28 a illustrates an embodiment of the invention where one firsttouches the screen at any point desired or comfortable, and said touchis temporarily indented into the screen. This indentation, for examplein a soft plastic screen, has the latency to later serve as reference totactilely come back to in a particular short session of touch commandactivity.

For example consider finger tip 2801 which has just contacted screen2810 at point “P, and left an indentation 2815 in the screen at thepoint in question (shown also in depth in section A—A). The purpose oftouching the screen in this instance was to hit a large projected boxicon 2820 displayed on the screen which cued the system to switch to anew screen.

As illustrated in FIG. 28 b, the new image such as 2850 displayed by thecomputer controlled projector, would have its control bar 2851 displayedsuch that the control lever 2860 would be at the same point “P” oftouch, that is at a point whose position was determined by the initialand variant touch location. Put another way, if you hit the big box(e.g. a heat icon) anywhere, the next control of a temperature bar isdisplayed where the starting point from its present temperature position(e.g. “74 degrees”) would be the same point “P” on the screen as youjust touched and indented. Thus one can come back to this point, withoutlooking at the screen at all, and move the lever left or right, toincrease or decrease temp from 74. The desired setting numbers can bedisplayed in large letters to make it easy to see.

In another instance it may be desired to come back to point P, and dragthe icon box to a new location toward the side of the display, whichtells the system a command depending on which side is desired (forexample up down, left right for mirror positions). The driver can dothis by feeling for the point previously touched—without taking his eyesoff the road. And he can further feel the edges of the display, oralternatively, ridges or other indentations in the display as taughtherein, if he wishes to explore the limits of the movement possible.

A useful display with such as screen is when the display is caused to“jump” to the point touched. This is discussed relative to FIG. 10 aswell.

The touch screen and related aspects of the invention open up new vistasfor adding useful functions for improved control or enjoyment. And thecompletely programmable nature of the device, provides a future abilityto insert new devices.

However, it is often difficult for persons to learn how to use thefunctionality deliverable. This is particularly true if the person hasjust gotten into a rental car say. Illustrated a useful embodiment ofthe invention, wherein the instructions for operation of a device (e.g.the heat control knob and associated computer display screen or screens)is displayed right along with the device on the tactile display or touchscreen of the invention.

For example consider in FIG. 1 or 2 for example a table of instructionsfor use can be down loaded from computer to image projector or otherdisplay mechanism and displayed right next the knob or slider orwhatever is being used. This makes the understanding much easier, thanfor example the simple storing of a manual in the computer for laterpresentation. This is particularly possible when the display is large,and the instructions, and the controls easy to see.

For example, as the person touches the knob, the instructions appear—ifan instruction mode is activated in the control computer. Theinstructions and knob are preferably large enough so that no difficultyis encountered seeing both together in one view.

This aspect of the invention is also very helpful with often complexfunctions. For example, when the knob is turned to heat as in FIG. 1 or2, and then pushed in, a thermal bar graph display for example isactivated which then itself is energized and used to set thetemperature. The instructions then for this function are subsequentlydisplayed to allow the user to at all times be informed as to what isdesired.

Further background information and commentary on the Automotive controlapplication.

The driver of the vehicle today, interacts with the controls of thevehicle primarily in three ways 1. by sight only 2. by touch/feel only(possibly accompanied by a sound, such as a click-corresponding to afeeling sensation in many cases, e.g. a detent) 3. by a quick glance,and then touch. Possibly followed by another quick glance to check afinal setting of a knob, slider control, etc.

Typically while driving, only methods 2 and 3 above can be effected, asthe control functions are too small to read (radio buttons arenotorious) and require too much concentration while driving. Indeed formany users, these functions are effectively completely inoperable whiledriving, as they cannot be worked by touch only due to the crowdednature of their placement, and they are too hard to see in aglance—especially for those who cannot easily correct their vision awayfrom the far sighted vision needed to drive.

Conversely, some functions, such as turn signal stalks, transmissionlevers, and some knobs are big enough and/or simple enough such thatonly touch is needed.

The addition of further desired functions such as internet communicationto such an already difficult situation, constitutes a major challenge,and is one of the issues addressed by this invention. For suchfunctions, designers today have heretofore envisioned only voicerecognition as the medium for actuation and data retrieval.

Problems Of Voice Recognition Alone

In theory one can simply talk to your car. You say wipers on, and theycome on, thanks to a program like IBM “via voice” or the like. But thishas great difficulty in execution. Noise, unnatural activity, and a needfor prompts in anything more than the simplest action, limit theeffectiveness. In addition, such action is sequential. You can only doone voice activated thing at a time, and often you must do this slowly.

Consider the extreme frustration of trying to carry out such actions ina noisy environment. You ask to retrieve emails. A voice prompt saysfrom which account, you speak a little softly, it then repeats therequest, asking you to speak more slowly/clearly saying it cant understand. All this takes your mind off the driving and any other taskneeded. In extreme examples one can become enraged at the system, not agood thing while driving. The more you try to do this way, the worse itis.

Consider too that it may not work well for another driver of thevehicle, as what it has learned with time may not match.

In addition to all of the above, there is a general shortage ofdashboard space on which to display data—be it from the vehiclesubsystems, or from external sources. Some types of data one would wishto display could be (see also discussion above):

Data Useful for Driving

Graphical data such as navigational maps, other Navigationalinformation;

Picture data useful for front and rear vision and other peripheralviewing;

Data Useful for Car Functions Maintenance issues Performance issues etc;

Data Useful for General Information stock price charts;

Hotel information;

Tourist information;

Data Useful for Specific Information;

Email messages for the user;

Emergency data from outside sources; or

Entertainment—such as TV, games, or internet for the passengers.

Tactile Displays and Touch Screens with Physical Attributes

A touch screen and/or tactile display such as disclosed above could beused to replace all the display and actuation functions of theconventional dashboard, while at the same time vastly augmenting thepotential scope of their application, as it can be reprogrammed to servedifferent needs in the vehicle at different times. One moment it can beused for heat and radio volume, the next for internet surfing, or cameraimage viewing (e.g. of the road behind the car) etc. The things to touchare programmably displayed.

Particularly of interest are functions which the touch screen or tactiledisplay may provide in a manner that is “Natural” to the user, such asknob turning, slider moving, switch rocking, turn signaling,transmission mode selection and the like.

The application and adoption by the public of such a radically differentdash approach, is probably aided by maintaining as much similarity aspossible to the dash of today. And the more tactile, the better, notonly for this purpose, but to allow control functions to be found andengaged with the least possible requirement to view the screen whiledriving in order to engage the needed functions.

The issue of “Glance” and degree thereof required

Lets consider the “Glance” needed, if any, of the driver to operatetoday's dash controls.

A worst case is the increasing need for added buttons and the like forthe ever multiplying electronic functions of the vehicle orcommunication systems. With fixed dash space, these become smaller andmore complex. And most are not easily arrayed within the drivers line ofsight.

In addition, many require not just a first glance, but a continued stareand often a squint—small radio buttons come to mind. Many accidents havebeen registered as a result, even though the little button once found,may be tactile in nature. Some functions even require one to stare atchanging numbers in an awkward location. Others require reading verysmall print (often impossible for those requiring glasses—a real dilemmafor middle aged and old particularly).

A touch screen can free up a whole expanse of screen to work with onefunction, for example a seat adjustment, or radio. Lettering can be bigenough to read without glasses. And by doing all functions this way, asaving in manufacturing cost results as well.

But the touch screen historically has a drawback as noted above, that ittoo may require of the driver more than a glance to operatesuccessfully. This is because there is nothing on a conventional touchscreen to reference ones self too—except the displayed data itself,which then has to be continually read. One means to do this by a “Headsup Display of sorts, displaying data on the windshield is disclosedherein which does not require one to take ones eyes of f the road.Another means is to use computer generated speech commands to forexample tell the driver that he has touched, or moved a touch commandbutton to a new position. In this case the Computer voice could say “74Degrees” for the new setting of the heat control for example.

Another approach is to create a touch screen or conventional displayhaving tactile features, a concept discussed in considerable detail inthe invention herein. It can thus be seen that the invention overcomesthe problems of conventional touch screens for automotive or otherapplications in several ways, which can even be combined in some cases.

A note on the viewing and reading of email and other messages.

If email can only be read or spoken, then output of email while drivinghas to be readable by driver in plain view. But this may be illegal inmost jurisdictions. Alternatively, the email message can be spoken tothe driver by a text to voice program in the car informationpresentation computer which also controls the displays of the invention.A suitable program for example is the Text-to-speech (TTS) program ofFonix Corporation.

The read back of email, is something that can be repeated as opportune,or safe. It also can just read who sent the message or other data, witha hand or finger control of the invention used to select whateverfurther info is desired from the computer and the email source.

A bigger problem on the input side, is for a driver to command actionsto a voice recognition program while trying to do something else (e.g.talk on a cell phone) or have conversation going with others. For thisfunction, a tactile based system, possibly combined with voice isconsidered superior.

Method of Manufacture and Sale

The invention herein would appear to be less costly to manufacture thanconventional dashboards, since two or three displays of the invention,which can be installed as modules along with the airbags (at least whereairbags are used), can contain all means necessary to control thevehicle, plus provide a large number of exciting further functions andattributes. This means that only a few connections need be made to theother wiring and computers in the car as well. Heating, defrosting andair conditioning ducts and other related HVAC devices such as mixingdoors etc are still required.

It is possible to assemble the displays/airbag modules by inserting themfrom the front (toward the inside of the car) in a finished vehicle.This keeps them from damage in the line type assembly process. From atrim point however, assembly from the rear allows perhaps a moreesthetically pleasing dash. Or the placement of a final trim panel overthe displays once assembled can be made.

Indeed the invention lends itself to dealer or after market installationas well. In one scenario, cars could be delivered to dealers withoutthese components, and customers would go into a special room at thedealership and chose their instrumentation to their own tastes. Thisthen would be programmed, any overlays or trim made, and installed inthe vehicle. This tends to build dealership traffic, which otherwise isthreatened due to the internet purchasing of vehicles trend, possible ifall vehicles are commodities.

It is noted that new display and touch functions can even be downloadedfrom an internet site such as that of the manufacturer—directly even tothe user. This allows a continuous upgrade path for approved“accessories” of a software nature.

The Problem of Driver Viewing of Prohibited Screens.

U.S. Pat. No. 5,949,345 assigned to Microsoft, addresses the issue ofwhat screens on a conventional computer display can be presented to thedriver. This is a useful concept to incorporate with the instantinvention, in this case making the total touch screen presentationgenerated in accordance with the ideas of U.S. Pat. No. 5,949,345 orthose herein.

There is another problem, that is how to sift through informationcontained in different screens? U.S. Pat. No. 5,995,3404 addresses onesolution to the problem of making the driver sequentially view differentscreens until he gets to the one of interest, and describes the manmachine interface devices such as joysticks needed to operate it. Thisreference describes a 3D screen presentation where all data is visible(supposedly) in one glance. This too can be incorporated with the touchscreen invention here, but is thought to be at least somewhatobjectionable as it departs from the classical presentation ofdashboards in cars, or instrument panels in planes, which other wise theinstant invention herein desirably resembles—a major advantage overpresentation of “Computer-like” displays.

Other Ideas

In the touch screen devices above, the various screen selections havebeen switched by either actuating a switch (which might be gestureactivated), or by voice or gesture command—that is say HEAT (or say,point at the heater), and the heat touch screen appears (or a screenhaving heat controls among other essential things, such as a speedometerdisplay).

One approach too is to have substantially every screen presented, suchas the myriad shown in FIG. 34, have a section common to all, whichcontains a selection control to switch screens.

The screens of the invention can as disclosed be used in numerouspositions on the dash or elsewhere, and in some cases can cover the neartotal width of the dash. It is possible to envision that the imagecomputers creating the displays for such screens could have multipleprograms stored in the vehicle, loaded in by CD ROMs or DVD Discs, ordown load new programs over the internet or other communication channelsfrom remote data bases.

Some novel screens could be for example to change the language of thescreen, English to Spanish for example. But not only that, but to changethe motif of the screen—to a more Spanish preferred layout if desired.OR the motif could depend on personal rather than regional tastes, forexample a dash with a GO BLUE theme, if the driver was a die hardUniversity of Michigan supporter for example. An Impressionist artlover, could have Van Goghs and Renoir's decorating his dash, and so on.In the extreme case of flexibility, not only could the language changeand the decoration change, but also the actual placement and type ofcontrols. For example ones standard and other screens of FIG. 34 couldbe changed to suit, limited by any laws that require certain data at alltimes, for example. Business interests could include stock tickers andthe like, with data received off the internet by cell phone.

And the display could be dynamic, with display of at least data varyingwith time, for example pictures varying with music

The same arguments also apply for the screen in ones home, or in onesoffice such as disclosed in FIGS. 19 and 20 for example.

It should be noted that when the sensing of a finger used to touch ascreen is performed independently of the screen itself, for example asdescribed in FIG. 6 c, then the programmable screen “feel can beanything which is transparent enough for the projected image, withoutregard to whether the screen itself has the ability itself to sensetouch.

The touch screen and other sensing aspects of the invention above areuseful for navigational purposes as well. A large screen presentation isavailable which is easy to see, and the driver can immediately designatea desired blow up of a route or other aspect on a map by rapidlytouching it. The driver can also quickly draw on the map a possiblealternative route, which the computer of the map system (local orremote) can then explore, and suggest possible avenues.

A note on New Business Models

In addition to the improved vehicle safety and control possibilitiesprovided by the invention, there are many “New Business” possibilitiesopened up by instrument panels build according to the invention. From aninternet business point of view, one for example, could customize onesown dash over the internet from the manufacturers or sellers web site,designing just how you wanted it presented. Perhaps you would want itall touch screen, or perhaps with some “real” knobs or the like. Thenwhen the car was delivered it would be special for you, also with yourname and address on the screen (a theft deterrent too!) for example. Ifyou wanted to change it later, it could be reprogrammed by the dealer(or possibly yourself), and if you wanted major change, you could goback to the web site and design another one which could be put in. Thiscould also done, just by changing an overlay and/the program forprojecting images (if a projection display) and interpreting the touchscreen and any other inputs such as cameras, microphones (for voiceinput or other purposes), etc.

It should be noted that the programmable tactile features, such as knobsof the invention can have their use changed by the user, or by downloadof new functions from some source such as a car company (g GM) web site.

Because the functions are shared, so much more data can bepresented—even down to minor things like giving heater outlet airblowing regimens and graphs—any data available indeed where one caneasily go back to important functions and data when needed.

The foregoing discussion should be understood as illustrative and shouldnot be considered to be limiting in any sense. While this invention hasbeen particularly shown and described with references to preferredembodiments thereof, it will be understood by those skilled in the artthat various changes in form and details may be made therein withoutdeparting from the spirit and scope of the invention.

The invention claimed is:
 1. A computer implemented method forperforming an operation on a computer system in response to inputreceived from a user, the method comprising: providing a touch screenadapted to display visually observable data and including a frontsurface having a tactile feature including at least one of a protrusionand an indentation, the tactile feature being integrally formed with thetouch screen front surface wherein the visually observable data isviewable through the tactile feature, the tactile feature beingreconfigurable to correspond to each of a plurality of vehicle controlfunctions; receiving a first touch input and a second touch input on thefront surface of the touch screen, wherein the first touch input and thesecond touch input are simultaneously in contact with the touch screen;determining a characteristic of the first touch input and the secondtouch input; and in response to said determining step, performing anoperation on the computer system to control one of the plurality ofvehicle control functions.
 2. The method according to claim 1 whereinthe characteristic includes the location of the first touch input andthe second touch input.
 3. The method according to claim 1 wherein: thefirst touch input includes at least one of a finger and a thumb incontact with the front surface of the touch screen; and the second touchinput includes at least one of a finger and a thumb in contact with thefront surface of the touch screen.
 4. The method according to claim 1wherein the tactile feature includes a key indent.
 5. The methodaccording to claim 4 further comprising displaying a virtual keyboardincluding a plurality of keys on the front surface, the key indentoverlying at least one of the plurality of keys.
 6. The method accordingto claim 1 wherein the tactile feature includes an annular recess. 7.The method according to claim 1 wherein the tactile feature includes anannular protrusion.
 8. The method according to claim 1 wherein thetactile feature includes a linear recess.
 9. The method according toclaim 1 wherein the tactile feature includes a linear protrusion. 10.The method according to claim 1 wherein the tactile feature includes around indentation.
 11. The method according to claim 1 wherein thetactile feature is convex protrusion adapted to change to a concaveindentation in response to a touch input on the tactile feature.
 12. Themethod according to claim 1 wherein: the first touch input generates afirst local distortion of the front surface; the second touch inputgenerates a second local distortion on the front surface; and thedetermining step is based in part on the first and second local surfacedistortion.
 13. The method according to claim 1 wherein: the touchscreen is adapted to display one of a plurality of data objectsproximate the tactile feature; and the operation is based on the dataobject proximate the tactile feature.
 14. The method according to claim1 wherein the tactile feature includes a protrusion defining a heightless than 0.01 inches with respect to an adjacent portion of the frontsurface of the touch screen to produce minimal discontinuity of anunderlying image displayed on the touch screen.
 15. The method accordingto claim 1 wherein the tactile feature includes an indentation defininga depth less than 0.01 inches with respect to an adjacent portion of thefront surface of the touch screen to produces a minimal discontinuity ofan image displayed on the touch screen.
 16. A method for performing anoperation on a computer system using a touch screen display, comprisingthe steps of: providing a touch screen display which can be touched by auser and is adapted to display visually observable data; providing onthe surface of the touch screen display one or more local disturbance orrelief features which can be felt by the user, the local disturbance orrelief features being integrally formed with the touch screen surface,wherein the visually observable data is viewable through the localdisturbance or relief feature, and wherein the at least one localdisturbance or relief feature is reconfigurable to correspond to aplurality of vehicle functions that are controllable by the user;receiving a touch input on the touch screen display proximate at leastone of the local disturbance or relief feature while providing a tactilesensation provided by the one or more features; sensing the location ofthe touch input; and controlling one of the plurality of vehiclefunctions using the sensed touch location on the touch screen display.17. A method according to claim 16 comprising the further step ofdisplaying graphics or other video images on the display.
 18. A methodaccording to claim 16 wherein the disturbance or relief feature does notsubstantially distort an image displayed.
 19. A method according toclaim 16 wherein images displayed on the display are not obscured by theuser.
 20. A method according to claim 16 wherein the disturbance orrelief feature is a bump.
 21. A method according to claim 16 wherein thedisturbance or relief feature is a ridge.
 22. A method according toclaim 16 wherein the disturbance or relief feature is an annular ring.23. A method according to claim 16 wherein the disturbance or relieffeature is a depression in the surface.
 24. A method according to claim16 wherein the disturbance or relief feature is a groove in the surface.25. A method according to claim 16 wherein the disturbance or relieffeature is coded.
 26. A method according to claim 16 wherein informationconcerning the vehicle function is displayed on or near the disturbanceor relief feature.
 27. A method according to claim 26 wherein theinformation is audio or climate information.